.. _tut_install_load_data: Toy example: Introduction and setup =================================== This page is a starting point of a series of tutorials that will help you get practical experience with FlexMeasures. Let's walk through an example from scratch! We'll ... - install FlexMeasures - create an account - load hourly prices What do you need? Your own computer, with one of two situations: either you have `Docker `_ or your computer supports Python 3.8+, pip and PostgresDB. The former might be easier, see the installation step below. But you choose. Below are the ``flexmeasures`` CLI commands we'll run, and which we'll explain step by step. There are some other crucial steps for installation and setup, so this becomes a complete example from scratch, but this is the meat: .. code-block:: bash # setup an account with a user, assets for battery & solar and an energy market (ID 1) $ flexmeasures add toy-account # load prices to optimise the schedule against $ flexmeasures add beliefs --sensor 1 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam Okay, let's get started! .. note:: You can copy the commands by hovering on the top right corner of code examples. You'll copy only the commands, not the output! Install Flexmeasures and the database --------------------------------------- .. tabs:: .. tab:: Docker If `docker `_ is running on your system, you're good to go. Otherwise, see `here `_. We start by installing the FlexMeasures platform, and then use Docker to run a postgres database and tell FlexMeasures to create all tables. .. code-block:: bash $ docker pull lfenergy/flexmeasures:latest $ docker pull postgres $ docker network create flexmeasures_network .. note:: A tip on Linux/macOS ― You might have the ``docker`` command, but need `sudo` rights to execute it. ``alias docker='sudo docker'`` enables you to still run this tutorial. After running these commands, we can start the Postgres database and the FlexMeasures app with the following commands: .. code-block:: bash $ docker run --rm --name flexmeasures-tutorial-db -e POSTGRES_PASSWORD=fm-db-passwd -e POSTGRES_DB=flexmeasures-db -d --network=flexmeasures_network postgres:latest $ docker run --rm --name flexmeasures-tutorial-fm --env SQLALCHEMY_DATABASE_URI=postgresql://postgres:fm-db-passwd@flexmeasures-tutorial-db:5432/flexmeasures-db --env SECRET_KEY=notsecret --env FLEXMEASURES_ENV=development --env LOGGING_LEVEL=INFO -d --network=flexmeasures_network -p 5000:5000 lfenergy/flexmeasures When the app has started, the FlexMeasures UI should be available at http://localhost:5000 in your browser. .. include:: ../notes/macOS-docker-port-note.rst To establish the FlexMeasures database structure, execute: .. code-block:: bash $ docker exec flexmeasures-tutorial-fm bash -c "flexmeasures db upgrade" Now - what's *very important* to remember is this: The rest of this tutorial will happen *inside* the ``flexmeasures-tutorial-fm`` container! This is how you hop inside the container and run a terminal there: .. code-block:: bash $ docker exec -it flexmeasures-tutorial-fm bash To leave the container session, hold CTRL-D or type "exit". To stop the containers, you can type .. code-block:: bash $ docker stop flexmeasures-tutorial-db $ docker stop flexmeasures-tutorial-fm To start the containers again, do this (note that re-running the `docker run` commands above *deletes and re-creates* all data!): .. code-block:: bash $ docker start flexmeasures-tutorial-db $ docker start flexmeasures-tutorial-fm .. note:: Got docker-compose? You could run this tutorial with 5 containers :) ― Go to :ref:`docker-compose-tutorial`. .. tab:: On your PC This example is from scratch, so we'll assume you have nothing prepared but a (Unix) computer with Python (3.8+) and two well-known developer tools, `pip `_ and `postgres `_. We'll create a database for FlexMeasures: .. code-block:: bash $ sudo -i -u postgres $ createdb -U postgres flexmeasures-db $ createuser --pwprompt -U postgres flexmeasures-user # enter your password, we'll use "fm-db-passwd" $ exit Then, we can install FlexMeasures itself, set some variables and tell FlexMeasures to create all tables: .. code-block:: bash $ pip install flexmeasures $ pip install highspy $ export SQLALCHEMY_DATABASE_URI="postgresql://flexmeasures-user:fm-db-passwd@localhost:5432/flexmeasures-db" SECRET_KEY=notsecret LOGGING_LEVEL="INFO" DEBUG=0 $ export FLEXMEASURES_ENV="development" $ flexmeasures db upgrade .. note:: When installing with ``pip``, on some platforms problems might come up (e.g. macOS, Windows). One reason is that FlexMeasures requires some libraries with lots of C code support (e.g. Numpy). One way out is to use Docker, which uses a prepared Linux image, so it'll definitely work. In case you want to re-run the tutorial, then it's recommended to delete the old database and create a fresh one. Run the following command to create a clean database with a new user, where it is optional. If you don't provide the user, then the default `postgres` user will be used to create the database. .. code-block:: bash $ make clean-db db_name=flexmeasures-db [db_user=flexmeasures] To start the web application, you can run: .. code-block:: bash $ flexmeasures run When started, the FlexMeasures UI should be available at http://localhost:5000 in your browser. .. include:: ../notes/macOS-port-note.rst Add some structural data --------------------------------------- The data we need for our example is both structural (e.g. a company account, a user, an asset) and numeric (we want market prices to optimize against). Let's create the structural data first. FlexMeasures offers a command to create a toy account with a battery: .. code-block:: bash $ flexmeasures add toy-account --kind battery Generic asset type `solar` created successfully. Generic asset type `wind` created successfully. Generic asset type `one-way_evse` created successfully. Generic asset type `two-way_evse` created successfully. Generic asset type `battery` created successfully. Generic asset type `building` created successfully. Generic asset type `process` created successfully. Creating account Toy Account ... Toy account Toy Account with user toy-user@flexmeasures.io created successfully. You might want to run `flexmeasures show account --id 1` Adding transmission zone type ... Adding NL transmission zone ... Created day-ahead prices The sensor recording day-ahead prices is day-ahead prices (ID: 1). Created Created discharging Created Created production The sensor recording battery discharging is discharging (ID: 2). The sensor recording solar forecasts is production (ID: 3). And with that, we're done with the structural data for this tutorial! If you want, you can inspect what you created: .. code-block:: bash $ flexmeasures show account --id 1 =========================== Account Toy Account (ID: 1) =========================== Account has no roles. All users: ID Name Email Last Login Last Seen Roles ---- -------- ------------------------ ------------ ----------- ------------- 1 toy-user toy-user@flexmeasures.io None None account-admin All assets: ID Name Type Location ---- ----------- ------- ----------------- 2 toy-building building (52.374, 4.88969) 3 toy-battery battery (52.374, 4.88969) 4 toy-solar solar (52.374, 4.88969) .. code-block:: bash $ flexmeasures show asset --id 2 ========================= Asset toy-building (ID: 2) ========================= Type Location Attributes ------- ----------------- ---------------------------- building (52.374, 4.88969) ==================================== Child assets of toy-building (ID: 2) ==================================== Id Name Type ------- ----------------- ---------------------------- 3 toy-battery battery 4 toy-solar solar No sensors in asset ... $ flexmeasures show asset --id 3 =================================== Asset toy-battery (ID: 3) Child of asset toy-building (ID: 2) =================================== Type Location Flex-Context Sensors to show Attributes ------- ----------------- -------------------------------- ------------------- ----------------------- battery (52.374, 4.88969) consumption-price: {'sensor': 1} Prices: [1] capacity_in_mw: 500 kVA Power flows: [3, 2] min_soc_in_mwh: 0.05 max_soc_in_mwh: 0.45 ==================================== Child assets of toy-battery (ID: 3) ==================================== No children assets ... All sensors in asset: ID Name Unit Resolution Timezone Attributes ---- ----------- ------ ------------ ---------------- ------------ 2 discharging MW 15 minutes Europe/Amsterdam Yes, that is quite a large battery :) You can also see that the asset has some meta information about its scheduling. Within :ref:`flex_context`, we noted where to find the relevant optimization signal for electricity consumption (Sensor 1, which stores day-ahead prices). .. note:: Obviously, you can use the ``flexmeasures`` command to create your own, custom account and assets. See :ref:`cli`. And to create, edit or read asset data via the API, see :ref:`v3_0`. We can also look at the battery asset in the UI of FlexMeasures (in Docker, the FlexMeasures web server already runs, on your PC you can start it with ``flexmeasures run``). Visit `http://localhost:5000/ `_ (username is "toy-user@flexmeasures.io", password is "toy-password"): .. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-schedule/asset-view-dashboard.png :align: center | .. note:: You won't see the map tiles, as we have not configured the :ref:`MAPBOX_ACCESS_TOKEN`. If you have one, you can configure it via ``flexmeasures.cfg`` (for Docker, see :ref:`docker_configuration`). .. _tut_toy_schedule_price_data: Add some price data --------------------------------------- Now to add price data. First, we'll create the CSV file with prices (EUR/MWh, see the setup for sensor 1 above) for tomorrow. .. code-block:: bash $ TOMORROW=$(date --date="next day" '+%Y-%m-%d') $ echo "Hour,Price $ ${TOMORROW}T00:00:00,10 $ ${TOMORROW}T01:00:00,11 $ ${TOMORROW}T02:00:00,12 $ ${TOMORROW}T03:00:00,15 $ ${TOMORROW}T04:00:00,18 $ ${TOMORROW}T05:00:00,17 $ ${TOMORROW}T06:00:00,10.5 $ ${TOMORROW}T07:00:00,9 $ ${TOMORROW}T08:00:00,9.5 $ ${TOMORROW}T09:00:00,9 $ ${TOMORROW}T10:00:00,8.5 $ ${TOMORROW}T11:00:00,10 $ ${TOMORROW}T12:00:00,8 $ ${TOMORROW}T13:00:00,5 $ ${TOMORROW}T14:00:00,4 $ ${TOMORROW}T15:00:00,4 $ ${TOMORROW}T16:00:00,5.5 $ ${TOMORROW}T17:00:00,8 $ ${TOMORROW}T18:00:00,12 $ ${TOMORROW}T19:00:00,13 $ ${TOMORROW}T20:00:00,14 $ ${TOMORROW}T21:00:00,12.5 $ ${TOMORROW}T22:00:00,10 $ ${TOMORROW}T23:00:00,7" > prices-tomorrow.csv This is time series data, in FlexMeasures we call *"beliefs"*. Beliefs can also be sent to FlexMeasures via API or imported from open data hubs like `ENTSO-E `_ or `OpenWeatherMap `_. However, in this tutorial we'll show how you can read data in from a CSV file. Sometimes that's just what you need :) .. code-block:: bash $ flexmeasures add beliefs --sensor 1 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam Successfully created beliefs In FlexMeasures, all beliefs have a data source. Here, we use the username of the user we created earlier. We could also pass a user ID, or the name of a new data source we want to use for CLI scripts. .. note:: Attention: We created and imported prices where the times have no time zone component! That happens a lot. FlexMeasures can localize them for you to a given timezone. Here, we localized the data to the timezone of the price sensor - ``Europe/Amsterdam`` - so the start time for the first price is `2022-03-03 00:00:00+01:00` (midnight in Amsterdam). Let's look at the price data we just loaded: .. code-block:: bash $ flexmeasures show beliefs --sensor 1 --start ${TOMORROW}T00:00:00+01:00 --duration PT24H Beliefs for Sensor 'day-ahead prices' (ID 1). Data spans a day and starts at 2022-03-03 00:00:00+01:00. The time resolution (x-axis) is an hour. ┌────────────────────────────────────────────────────────────┐ │ ▗▀▚▖ │ │ ▗▘ ▝▖ │ │ ▞ ▌ │ │ ▟ ▐ │ 15EUR/MWh │ ▗▘ ▝▖ ▗ │ │ ▗▘ ▚ ▄▞▘▚▖ │ │ ▞ ▐ ▄▀▘ ▝▄ │ │ ▄▞ ▌ ▛ ▖ │ │▀ ▚ ▐ ▝▖ │ │ ▝▚ ▖ ▗▘ ▝▖ │ 10EUR/MWh │ ▀▄▄▞▀▄▄ ▗▀▝▖ ▞ ▐ │ │ ▀▀▜▘ ▝▚ ▗▘ ▚ │ │ ▌ ▞ ▌│ │ ▝▖ ▞ ▝│ │ ▐ ▞ │ │ ▚ ▗▞ │ 5EUR/MWh │ ▀▚▄▄▄▄▘ │ └────────────────────────────────────────────────────────────┘ 5 10 15 20 ██ day-ahead prices Again, we can also view these prices in the `FlexMeasures UI `_: .. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-schedule/sensor-data-prices.png :align: center | .. note:: Technically, these prices for tomorrow may be forecasts (depending on whether you are running through this tutorial before or after the day-ahead market's gate closure). You can also use FlexMeasures to compute forecasts yourself. See :ref:`tut_forecasting_scheduling`.