setup-and-guidelines.rst 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322
  1. .. _developing:
  2. Developing for FlexMeasures
  3. ===========================
  4. This page instructs developers who work on FlexMeasures how to set up the development environment.
  5. Furthermore, we discuss several guidelines and best practices.
  6. .. contents:: Table of contents
  7. :local:
  8. :depth: 1
  9. |
  10. .. note:: Are you implementing code based on FlexMeasures, you're probably interested in :ref:`datamodel`.
  11. Getting started
  12. ------------------
  13. Virtual environment
  14. ^^^^^^^^^^^^^^^^^^^^
  15. Using a virtual environment is best practice for Python developers. We also strongly recommend using a dedicated one for your work on FlexMeasures, as our make target (see below) will use ``pip-sync`` to install dependencies, which could interfere with some libraries you already have installed.
  16. * Make a virtual environment: ``python3.10 -m venv flexmeasures-venv`` or use a different tool like ``mkvirtualenv`` or virtualenvwrapper. You can also use
  17. an `Anaconda distribution <https://conda.io/docs/user-guide/tasks/manage-environments.html>`_ as base with ``conda create -n flexmeasures-venv python=3.10``.
  18. * Activate it, e.g.: ``source flexmeasures-venv/bin/activate``
  19. Download FlexMeasures
  20. ^^^^^^^^^^^^^^^^^^^^^^^
  21. Clone the `FlexMeasures repository <https://github.com/FlexMeasures/flexmeasures.git>`_ from GitHub.
  22. .. code-block:: bash
  23. $ git clone https://github.com/FlexMeasures/flexmeasures.git
  24. Dependencies
  25. ^^^^^^^^^^^^^^^^^^^^
  26. Go into the ``flexmeasures`` folder and install all dependencies including the ones needed for development:
  27. .. code-block:: bash
  28. $ cd flexmeasures
  29. $ make install-for-dev
  30. :ref:`Install the LP solver <install-lp-solver>`. On Linux, the HiGHS solver can be installed with:
  31. .. code-block:: bash
  32. $ pip install highspy
  33. On MacOS it will be installed locally by `make install-for-test` and no actions are required on your part
  34. Besides highs, the CBC solver is required for tests as well:
  35. .. tabs::
  36. .. tab:: Linux
  37. .. code-block:: bash
  38. $ apt-get install coinor-cbc
  39. .. tab:: MacOS
  40. .. code-block:: bash
  41. $ brew install cbc
  42. Configuration
  43. ^^^^^^^^^^^^^^^^^^^^
  44. Most configuration happens in a config file, see :ref:`configuration` on where it can live and all supported settings.
  45. For now, we let it live in your home directory and we add the first required setting: a secret key:
  46. .. code-block:: bash
  47. echo "SECRET_KEY=\"`python3 -c 'import secrets; print(secrets.token_hex(24))'`\"" >> ~/.flexmeasures.cfg
  48. Also, we add some env settings in an `.env` file. Create that file in the `flexmeasures` directory (from where you'll run flexmeasures) and enter:
  49. .. code-block:: bash
  50. FLEXMEASURES_ENV="development"
  51. LOGGING_LEVEL="INFO"
  52. The development mode makes sure we don't need SSL to connect, among other things.
  53. Database
  54. ^^^^^^^^^^^^^^^^
  55. See :ref:`host-data` for tips on how to install and upgrade databases (postgres and redis).
  56. Loading data
  57. ^^^^^^^^^^^^^^^^^^^^
  58. If you have a SQL Dump file, you can load that:
  59. .. code-block:: bash
  60. $ psql -U {user_name} -h {host_name} -d {database_name} -f {file_path}
  61. One other possibility is to add a toy account (which owns some assets and a battery):
  62. .. code-block:: bash
  63. $ flexmeasures add toy-account
  64. Run locally
  65. ^^^^^^^^^^^^^^^^^^^^
  66. Now, to start the web application, you can run:
  67. .. code-block:: bash
  68. $ flexmeasures run
  69. Or:
  70. .. code-block:: bash
  71. $ python run-local.py
  72. And access the server at http://localhost:5000
  73. If you added a toy account, you could log in with `toy-user@flexmeasures.io`, password `toy-password`.
  74. Otherwise, you need to add some other user first. Here is how we add an admin:
  75. .. code-block:: bash
  76. $ flexmeasures add account --name MyCompany
  77. $ flexmeasures add user --username admin --account 1 --email admin@mycompany.io --roles admin
  78. (The `account` you need in the 2nd command is printed by the 1st)
  79. .. include:: ../notes/macOS-port-note.rst
  80. .. note::
  81. If you are on Windows, then running & developing FlexMeasures will not work 100%. For instance, the queueing only works if you install rq-win (https://github.com/michaelbrooks/rq-win) manually and the make tooling is difficult to get to work as well.
  82. We recommend to use the Windows Sub-system for Linux (https://learn.microsoft.com/en-us/windows/wsl/install) or work via Docker-compose (https://flexmeasures.readthedocs.io/en/latest/dev/docker-compose.html).
  83. Logfile
  84. --------
  85. FlexMeasures logs to a file called ``flexmeasures.log``. You'll find this in the application's context folder, e.g. where you called ``flexmeasures run``.
  86. A rolling log file handler is used, so if ``flexmeasures.log`` gets to a few megabytes in size, it is copied to `flexmeasures.log.1` and the original file starts over empty again.
  87. The default logging level is ``WARNING``. To see more, you can update this with the config setting ``LOGGING_LEVEL``, e.g. to ``INFO`` or ``DEBUG``
  88. Mocking an Email Server for Development
  89. --------------------------------
  90. To handle emails locally during development, you can use MailHog. Follow these steps to set it up:
  91. .. code-block:: bash
  92. $ docker run -p 8025:8025 -p 1025:1025 --name mailhog mailhog/mailhog
  93. $ export MAIL_PORT=1025 # You can also add this to your local flexmeasures.cfg
  94. Now, emails (e.g., password-reset) are being sent via this local server. Go to http://localhost:8025 to see all sent emails in a web UI.
  95. Tests
  96. -----
  97. You can run automated tests with:
  98. .. code-block:: bash
  99. $ make test
  100. which behind the curtains installs dependencies and calls ``pytest``.
  101. However, a test database (postgres) is needed to run these tests. If you have postgres, here is the short version on how to add the test database:
  102. .. code-block:: bash
  103. $ make clean-db db_name=flexmeasures_test db_user=flexmeasures_test
  104. $ # the password for the db user is "flexmeasures_test"
  105. .. note:: The section :ref:`host-data` has more details on using postgres for FlexMeasures.
  106. Alternatively, if you don't feel like installing postgres for the time being, here is a docker command to provide a test database:
  107. .. code-block:: bash
  108. $ docker run --rm --name flexmeasures-test-db -e POSTGRES_PASSWORD=flexmeasures_test -e POSTGRES_DB=flexmeasures_test -e POSTGRES_USER=flexmeasures_test -p 5432:5432 -v ./ci/load-psql-extensions.sql:/docker-entrypoint-initdb.d/load-psql-extensions.sql -d postgres:latest
  109. .. warning:: This assumes that the port 5432 is not being used on your machine (for instance by an existing postgres database service).
  110. If you want the tests to create a coverage report (printed on the terminal), you can run the ``pytest`` command like this:
  111. .. code-block:: bash
  112. $ pytest --cov=flexmeasures --cov-config .coveragerc
  113. You can add `--cov-report=html`, after which a file called `htmlcov/index.html` is generated.
  114. Or, after a test run with coverage turned on as shown above, you can still generate it in another form:
  115. .. code-block:: bash
  116. $ python3 -m coverage [html|lcov|json]
  117. Versioning
  118. ----------
  119. We use `setuptool_scm <https://github.com/pypa/setuptools_scm/>`_ for versioning, which bases the FlexMeasures version on the latest git tag and the commits since then.
  120. So as a developer, it's crucial to use git tags for versions only.
  121. We use semantic versioning, and we always include the patch version, not only max and min, so that setuptools_scm makes the correct guess about the next minor version. Thus, we should use ``2.0.0`` instead of ``2.0``.
  122. See ``to_pypi.sh`` for more commentary on the development versions.
  123. Our API has its own version, which moves much slower. This is important to explicitly support outside apps who were coded against older versions.
  124. Auto-applying formatting and code style suggestions
  125. -----------------------------------------------------
  126. We use `Black <https://github.com/ambv/black>`_ to format our Python code and `Flake8 <https://flake8.pycqa.org>`_ to enforce the PEP8 style guide and linting.
  127. We also run `mypy <http://mypy-lang.org/>`_ on many files to do some static type checking.
  128. We do this so real problems are found faster and the discussion about formatting is limited.
  129. All of these can be installed by using ``pip``, but we recommend using them as a pre-commit hook. To activate that behaviour, do:
  130. .. code-block:: bash
  131. $ pip install pre-commit
  132. $ pre-commit install
  133. in your virtual environment.
  134. Now each git commit will first run ``flake8``, then ``black`` and finally ``mypy`` over the files affected by the commit
  135. (\ ``pre-commit`` will install these tools into its own structure on the first run).
  136. This is also what happens automatically server-side when code is committed to a branch (via GitHub Actions), but having those tests locally as well will help you spot these issues faster.
  137. If ``flake8``, ``black`` or ``mypy`` propose changes to any file, the commit is aborted (saying that it "failed").
  138. The changes proposed by ``black`` are implemented automatically (you can review them with `git diff`). Some of them might even resolve the ``flake8`` warnings :)
  139. Using Visual Studio, including spell checking
  140. ----------------------------------------------
  141. Are you using Visual Studio Code? Then the code you just cloned also contains the editor configuration (part of) our team is using (see `.vscode`)!
  142. We recommend installing the flake8 and spellright extensions.
  143. For spellright, the FlexMeasures repository contains the project dictionary. Here are steps to link main dictionaries, which usually work on a Linux system:
  144. .. code-block:: bash
  145. $ mkdir $HOME/.config/Code/Dictionaries
  146. $ ln -s /usr/share/hunspell/* ~/.config/Code/Dictionaries
  147. Consult the extension's Readme for other systems.
  148. A hint about using notebooks
  149. ---------------
  150. If you edit notebooks, make sure results do not end up in git:
  151. .. code-block:: bash
  152. $ conda install -c conda-forge nbstripout
  153. $ nbstripout --install
  154. (on Windows, maybe you need to look closer at https://github.com/kynan/nbstripout)
  155. A hint for Unix developers
  156. --------------------------------
  157. I added this to my ~/.bashrc, so I only need to type ``fm`` to get started and have the ssh agent set up, as well as up-to-date code and dependencies in place.
  158. .. code-block:: bash
  159. addssh(){
  160. eval `ssh-agent -s`
  161. ssh-add ~/.ssh/id_github
  162. }
  163. fm(){
  164. addssh
  165. cd ~/workspace/flexmeasures
  166. git pull # do not use if any production-like app runs from the git code
  167. workon flexmeasures-venv # this depends on how you created your virtual environment
  168. make install-for-dev
  169. }
  170. .. note:: All paths depend on your local environment, of course.