# LHC Signal Monitoring API Package
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/logo.png" width=25%>

This is a package with an API for signal query and processing for the Signal Monitoring project.

## Signal Monitoring Project Architecture

The Signal Monitoring project architecture consists of four elements:

1. API for logging db query and signal processing (<a href="https://gitlab.cern.ch/lhcdata/lhc-sm-api">https://gitlab.cern.ch/lhcdata/lhc-sm-api</a>)
2. Signal Monitoring notebooks (<a href="https://gitlab.cern.ch/lhcdata/lhc-sm-apps">https://gitlab.cern.ch/lhcdata/lhc-sm-apps</a>)
3. HWC and Operation notebooks (<a href="https://gitlab.cern.ch/lhcdata/lhc-sm-hwc">https://gitlab.cern.ch/lhcdata/lhc-sm-hwc</a>)
4. Scheduler for execution of HWC notebooks and monitoring applications (<a href="https://gitlab.cern.ch/lhcdata/lhc-sm-scheduler">https://gitlab.cern.ch/lhcdata/lhc-sm-scheduler</a>)
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/lhc-sm-architecture.png" width=50%>

## Signal Monitoring API Architecture


The API is a collection of six modules for:  
- database access (DBSignal); 
- signal, system, circuit naming (Metadata); 
- signal and event references (Reference); 
- embedded domain specific language for signal/feature/event query and processing, signal assertion, and feature engineering;
- signal query, analysis, and plot (Analysis); 
- graphical user interfaces for browsing of historicalsignal features (GUI).

<img src="https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/lhc-sm-api-architecture.png" width=50%>

## Installation
In order to use the API, it has to be installed with a python package installer as

```python
pip install --user lhcsmapi
```
Check the latest version at <a href="https://pypi.org/project/lhcsmapi/">https://pypi.org/project/lhcsmapi/</a>

The API relies on several external python packages which have to be installed in a similar manner. The list of packages is stored in the <u><i>requirements.txt</i></u> file.

If you use SWAN, the service provides a set of pre-installed python packages through CVMFS. The LHC-SM notebooks require installation of several additional packages on top of CVMFS. In order to install a package, please open a SWAN Terminal by clicking [>_] icon in the top right corner.

![SWAN CLI Button](https://gitlab.cern.ch/LHCData/lhc-sm-hwc/-/raw/master/figures/swan-cli-button.png)

Five additional python packages have to be installed:
- tzlocal - for time zone convertion
- tqdm - for progress bar to track queries
- influxdb - for communication with an Influxdb
- plotly - for interactive plotting of circuit schematics 
- lhcsmapi - for LHC-SM API
In order to install a package please execute the following command
```
$ pip install --user package_name
```

The expected output, after installing all packages, is presented in five figures below.

- SWAN Terminal output after successful installation of tzlocal package.
![SWAN pip install tzlocal](https://gitlab.cern.ch/LHCData/lhc-sm-hwc/-/raw/master/figures/swan-pip-install-tzlocal.png)

- SWAN Terminal output after successful installation of tqdm package.
![SWAN pip install tqdm](https://gitlab.cern.ch/LHCData/lhc-sm-hwc/-/raw/master/figures/swan-pip-install-tqdm.png)

- SWAN Terminal output after successful installation of influxdb package.
![SWAN pip install influxdb](https://gitlab.cern.ch/LHCData/lhc-sm-hwc/-/raw/master/figures/swan-pip-install-influxdb.png)

- SWAN Terminal output after successful installation of plotly package.
![SWAN pip install plotly](https://gitlab.cern.ch/LHCData/lhc-sm-hwc/-/raw/master/figures/swan-pip-install-plotly.png)

- SWAN Terminal output after successful installation of lhcsmapi package.
![SWAN pip install lhcsmapi](https://gitlab.cern.ch/LHCData/lhc-sm-hwc/-/raw/master/figures/swan-pip-install-lhcsmapi.png)

**Updating lhcsmapi Package**

Please note that the first four packages (tzlocal, tqdm, influxdb, plotly) have to be installed only once while the last one is still in the development phase and subject to frequent updates. Please send us an e-mail request (mailto:lhc-signal-monitoring@cern.ch) if you want to subscribe for updates. In order to update the lhcsmapi package, please execute the following command.
```
$ pip install --user --upgrade lhcsmapi
```

In case this command returns an error, please try to execute it again. Should that operation also fail, please uninstall the package by executing
```
$ pip uninstall lhcsmapi
```

and performing a fresh installation the package
```
$ pip install --user lhcsmapi
```

Should you experience any further issues with installing a package, please contact <a href="https://swan.web.cern.ch">SWAN support</a>.


# 1. Time
**Time** class encapsulates provides a set of methods for conversion between various time formats. In addition it provides a generator for days between given periods. Main methods are:
- Time.to_unix_timestamp(input, unit='ns')
- Time.to_pandas_timestamp(input, unit='ns')
- Time.to_string(input, unit='ns')
- Time.to_datetime(input, unit='ns')

The unit argument is optional. Its default value is 'ns', i.e., nanosecond.

## 1.1. Time.to_unix_timestamp(input, unit='ns')
This method is useful to obtain a timestamp to query PM database


```python
from lhcsmapi.Time import Time
date_time_str = "2018-05-01 00:00:00"
Time.to_unix_timestamp(date_time_str, unit="ns")
```




    1525125600000000000




```python
from lhcsmapi.Time import Time
from datetime import datetime
date_time_dt = datetime(2018, 5, 1, 0, 0, 0, 0)
Time.to_unix_timestamp(date_time_dt, unit="ns")
```




    1525125600000000000




```python
from lhcsmapi.Time import Time
timestamp = 1525125600000000000
Time.to_unix_timestamp(timestamp, unit="ns")
```




    1525125600000000000



Any method also works without the unit. The default unit is ns.


```python
from lhcsmapi.Time import Time
timestamp = 1525125600000000000
Time.to_unix_timestamp(timestamp)
```




    1525125600000000000




```python
from lhcsmapi.Time import Time
import pandas as pd
timestamp_pd = pd.Timestamp(2018, 5, 1, 0)
Time.to_unix_timestamp(timestamp_pd, unit="ns")
```




    1525125600000000000



## 1.2. Time.to_pandas_timestamp(input, unit='ns')


```python
from lhcsmapi.Time import Time
date_time_str = "2019-07-10 10:23:27.013"
Time.to_pandas_timestamp(date_time_str, unit="ns")
```




    Timestamp('2019-07-10 10:23:27.013000+0200', tz='Europe/Zurich')




```python
from lhcsmapi.Time import Time
from datetime import datetime
date_time_dt = datetime(2019, 7, 10, 10, 23, 27, 13)
Time.to_pandas_timestamp(date_time_dt, unit="ns")
```




    Timestamp('2019-07-10 10:23:27.000013+0200', tz='Europe/Zurich')




```python
from lhcsmapi.Time import Time
timestamp = 1562747007013000000
Time.to_pandas_timestamp(timestamp, unit="ns")
```




    Timestamp('2019-07-10 10:23:27.013000+0200', tz='Europe/Zurich')




```python
from lhcsmapi.Time import Time
import pandas as pd
timestamp_pd = pd.Timestamp(2019, 7, 10, 10, 23, 27, 13000)
Time.to_pandas_timestamp(timestamp_pd, unit="ns")
```




    Timestamp('2019-07-10 10:23:27.013000+0200', tz='Europe/Zurich')



## 1.3. Time.to_string(input, unit='ns')


```python
from lhcsmapi.Time import Time
date_time_str = "2017-07-02 01:23:27"
Time.to_string(date_time_str, unit="ns")
```




    '2017-07-02 01:23:27+02:00'




```python
from lhcsmapi.Time import Time
from datetime import datetime
date_time_dt = datetime(2017, 7, 2, 1, 23, 27, 0)
Time.to_string(date_time_dt, unit="ns")
```




    '2017-07-02 01:23:27+02:00'




```python
from lhcsmapi.Time import Time
timestamp = 1498951407000000000
Time.to_string(timestamp, unit="ns")
```




    '2017-07-02 01:23:27+02:00'




```python
from lhcsmapi.Time import Time
import pandas as pd
timestamp_pd = pd.Timestamp(2018, 5, 1, 1, 23, 27)
Time.to_string(timestamp_pd, unit="ns")
```




    '2018-05-01 01:23:27+02:00'



There is also a version of this method returning a string without the time zone


```python
from lhcsmapi.Time import Time
import pandas as pd
timestamp_pd = pd.Timestamp(2018, 5, 1, 1, 23, 27)
Time.to_string_short(timestamp_pd, unit="ns")
```




    '2018-05-01 01:23:27'



## 1.4. Time.to_datetime(input, unit='ns')


```python
from lhcsmapi.Time import Time
date_time_str = "2019-01-02 23:59:11"
Time.to_datetime(date_time_str, unit="ns")
```




    datetime.datetime(2019, 1, 2, 23, 59, 11, tzinfo=<DstTzInfo 'Europe/Zurich' CET+1:00:00 STD>)




```python
from lhcsmapi.Time import Time
from datetime import datetime
date_time_dt = datetime(2019, 1, 2, 23, 59, 11, 0)
Time.to_datetime(date_time_dt, unit="ns")
```




    datetime.datetime(2019, 1, 2, 23, 59, 11, tzinfo=<DstTzInfo 'Europe/Zurich' CET+1:00:00 STD>)




```python
from lhcsmapi.Time import Time
timestamp = 1546469951000000000
Time.to_datetime(timestamp, unit="ns")
```




    datetime.datetime(2019, 1, 2, 23, 59, 11, tzinfo=<DstTzInfo 'Europe/Zurich' CET+1:00:00 STD>)




```python
from lhcsmapi.Time import Time
import pandas as pd
timestamp_pd = pd.Timestamp(2019, 1, 2, 23, 59, 11, 0)
Time.to_datetime(timestamp_pd, unit="ns")
```




    datetime.datetime(2019, 1, 2, 23, 59, 11, tzinfo=<DstTzInfo 'Europe/Zurich' CET+1:00:00 STD>)



## 1.5. Time.to_unix_timestamp_in_sec(input, tz)


```python
from lhcsmapi.Time import Time
timestamp = 1426220469491000000
Time.to_unix_timestamp_in_sec(timestamp)
```




    1426220469.4910002



## 1.6. Time.get_query_period_in_unix_time(t_start, t_end, duration, tz)

There are three ways of executing this method:
- start time and end time


```python
from lhcsmapi.Time import Time
time_date_start = 1426220459491000000
time_date_end = 1426220479491000000
Time.get_query_period_in_unix_time(time_date_start, time_date_end, None, tz='Europe/Zurich')
```




    (1426220459491000000, 1426220479491000000)



- start time, duration with a single element list to indicate end time after the start time


```python
from lhcsmapi.Time import Time
time_date_start = 1426220459491000000
duration = [(20, 's')]
Time.get_query_period_in_unix_time(time_date_start, None, duration, tz='Europe/Zurich')
```




    (1426220459491000000, 1426220479491000000)



- start time, duration with a two element list to indicate end time befor (1st element) and after (2nd element) the start time


```python
from lhcsmapi.Time import Time
time_date_start = 1426220469491000000
duration =  [(10, 's'), (10, 's')]
Time.get_query_period_in_unix_time(time_date_start, None, duration, tz='Europe/Zurich')
```




    (1426220459491000000, 1426220479491000000)



## 1.7. Time.daterange(time_date_start, time_date_end)
Useful to iterate over days while e.g. sequentially querying PM database for events (there is a limit of 24h query duration in this case).


```python
from datetime import date
time_date_start = date(2014, 1, 1)
time_date_end = date(2014, 1, 3)
for single_date in Time.daterange(time_date_start, time_date_end):
    print(single_date)
```

    2014-01-01
    2014-01-02


In case the start date is later than the end date, the generator does not return output


```python
from datetime import date
time_date_start = date(2014, 1, 5)
time_date_end = date(2014, 1, 3)
for single_date in Time.daterange(time_date_start, time_date_end):
    print(single_date)
```

# 2. Timer
**Timer** class provides a base functionality for measuring code execution time.


```python
from lhcsmapi.Timer import Timer
t = Timer()

with Timer():
    for i in range(1000000):
        i = i+1
```

    Elapsed: 0.178 s.


equivalent to


```python
import time
t = time.time()
for i in range(1000000):
        i = i+1
time.time() - t
```




    0.1734294891357422



# 3. Metadata
The Metadata (lhcsmapi.metadata) module contains methods to retrieve various signal and circuit names.  
In order to avoid storing full names and enable updating signal names based on time. A signal hierarchy is encoded with a dictionary and can be accessed through links in the table below

|Circuit type|Hyperlink|
|------------|---------|
|RB          |https://gitlab.cern.ch/LHCData/lhc-sm-api/blob/master/lhcsmapi/metadata/RB_METADATA.json|
|RQ          |https://gitlab.cern.ch/LHCData/lhc-sm-api/blob/master/lhcsmapi/metadata/RQ_METADATA.json|
|IT          |https://gitlab.cern.ch/LHCData/lhc-sm-api/blob/master/lhcsmapi/metadata/IT_METADATA.json|
|IPQ2        |https://gitlab.cern.ch/LHCData/lhc-sm-api/blob/master/lhcsmapi/metadata/IPQ2_METADATA.json|
|IPQ4        |https://gitlab.cern.ch/LHCData/lhc-sm-api/blob/master/lhcsmapi/metadata/IPQ4_METADATA.json|
|IPQ8        |https://gitlab.cern.ch/LHCData/lhc-sm-api/blob/master/lhcsmapi/metadata/IPQ8_METADATA.json|
|IPD2        |https://gitlab.cern.ch/LHCData/lhc-sm-api/blob/master/lhcsmapi/metadata/IPD2_METADATA.json|
|IPD2_B1B2   |https://gitlab.cern.ch/LHCData/lhc-sm-api/blob/master/lhcsmapi/metadata/IPD2_B1B2_METADATA.json|
|60A         |https://gitlab.cern.ch/LHCData/lhc-sm-api/blob/master/lhcsmapi/metadata/60A_METADATA.json|
|80-120A     |https://gitlab.cern.ch/LHCData/lhc-sm-api/blob/master/lhcsmapi/metadata/80-120A_METADATA.json|

### 3.1. Signal Metadata 
**SignalMetadata** class stores information about signal and circuit names as well as corresponding metadata.

- get beam mode description


```python
from lhcsmapi.metadata.SignalMetadata import SignalMetadata

SignalMetadata.get_beam_mode_details()
```




<div>

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Mode</th>
      <th>Name</th>
      <th>Description</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>NOMODE</td>
      <td>No mode, data is not available, not set</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>SETUP</td>
      <td>Setup</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>INJPILOT</td>
      <td>Pilot injection</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4</td>
      <td>INJINTR</td>
      <td>Intermediate injection</td>
    </tr>
    <tr>
      <th>4</th>
      <td>5</td>
      <td>INJNOMN</td>
      <td>Nominal injection</td>
    </tr>
    <tr>
      <th>5</th>
      <td>6</td>
      <td>PRERAMP</td>
      <td>Before ramp</td>
    </tr>
    <tr>
      <th>6</th>
      <td>7</td>
      <td>RAMP</td>
      <td>Ramp</td>
    </tr>
    <tr>
      <th>7</th>
      <td>8</td>
      <td>FLATTOP</td>
      <td>Flat top</td>
    </tr>
    <tr>
      <th>8</th>
      <td>9</td>
      <td>SQUEEZE</td>
      <td>Squeeze</td>
    </tr>
    <tr>
      <th>9</th>
      <td>10</td>
      <td>ADJUST</td>
      <td>Adjust beam on flat top</td>
    </tr>
    <tr>
      <th>10</th>
      <td>11</td>
      <td>STABLE</td>
      <td>Stable beam for physics</td>
    </tr>
    <tr>
      <th>11</th>
      <td>12</td>
      <td>UNSTABLE</td>
      <td>Unstable beam</td>
    </tr>
    <tr>
      <th>12</th>
      <td>13</td>
      <td>BEAMDUMP</td>
      <td>Beam dump</td>
    </tr>
    <tr>
      <th>13</th>
      <td>14</td>
      <td>RAMPDOWN</td>
      <td>Ramp down</td>
    </tr>
    <tr>
      <th>14</th>
      <td>15</td>
      <td>RECOVERY</td>
      <td>Recovering</td>
    </tr>
    <tr>
      <th>15</th>
      <td>16</td>
      <td>INJDUMP</td>
      <td>Inject and dump</td>
    </tr>
    <tr>
      <th>16</th>
      <td>17</td>
      <td>CIRCDUMP</td>
      <td>Circulate and dump</td>
    </tr>
    <tr>
      <th>17</th>
      <td>18</td>
      <td>ABORT</td>
      <td>Recovery after a beam permit flag drop</td>
    </tr>
    <tr>
      <th>18</th>
      <td>19</td>
      <td>CYCLING</td>
      <td>Pre-cycle before injection, no beam</td>
    </tr>
    <tr>
      <th>19</th>
      <td>20</td>
      <td>WBDUMP</td>
      <td>Warning beam dump</td>
    </tr>
    <tr>
      <th>20</th>
      <td>21</td>
      <td>NOBEAM</td>
      <td>No beam or preparation for beam</td>
    </tr>
  </tbody>
</table>
</div>



- get circuit types


```python
from lhcsmapi.metadata.SignalMetadata import SignalMetadata

SignalMetadata.get_circuit_types()
```




    ['RB',
     'RQ',
     'IT',
     'IPD2_B1B2',
     'IPD2',
     'IPQ2',
     'IPQ4',
     'IPQ8',
     '600A',
     '60A',
     '80-120A']



- get circuit names


```python
from lhcsmapi.metadata.SignalMetadata import SignalMetadata

SignalMetadata.get_circuit_names_or_types('RB')
```




    ['RB.A12',
     'RB.A23',
     'RB.A34',
     'RB.A45',
     'RB.A56',
     'RB.A67',
     'RB.A78',
     'RB.A81']



- get signal name for PM


```python
from lhcsmapi.metadata.SignalMetadata import SignalMetadata

SignalMetadata.get_signal_name('RB', 'RB.A12', 'PC', 'PM', 'I_MEAS')
```




    'STATUS.I_MEAS'



- get signal name for CALS


```python
from lhcsmapi.metadata.SignalMetadata import SignalMetadata

SignalMetadata.get_signal_name('RB', 'RB.A12', 'PC', 'CALS', 'I_MEAS')
```




    'RPTE.UA23.RB.A12:I_MEAS'



- get signal name for NXCALS


```python
from lhcsmapi.metadata.SignalMetadata import SignalMetadata

SignalMetadata.get_signal_name('RB', 'RB.A12', 'PC', 'NXCALS', 'I_MEAS')
```




    'I_MEAS'



- get metadata for PM


```python
from lhcsmapi.metadata.SignalMetadata import SignalMetadata

SignalMetadata.get_circuit_signal_database_metadata('RB', 'RB.A12', 'PC', 'PM')
```




    {'system': 'FGC',
     'source': 'RPTE.UA23.RB.A12',
     'className': '51_self_pmd',
     'I_REF': 'STATUS.I_REF',
     'I_MEAS': 'STATUS.I_MEAS',
     'V_REF': 'STATUS.V_REF',
     'V_MEAS': 'STATUS.V_MEAS',
     'I_EARTH': 'IEARTH.IEARTH',
     'I_EARTH_PCNT': 'STATUS.I_EARTH_PCNT',
     'I_A': 'IAB.I_A',
     'I_B': 'IAB.I_B'}



- get metadata for NXCALS


```python
from lhcsmapi.metadata.SignalMetadata import SignalMetadata

SignalMetadata.get_circuit_signal_database_metadata('RB', 'RB.A12', 'PC', 'NXCALS')
```




    {'system': 'CMW',
     'device': 'RPTE.UA23.RB.A12',
     'property': 'SUB',
     'I_MEAS': 'I_MEAS'}



### 3.2. Mapping Metadata
Some signal names obtained with SignalMetadata functions have a wildcard in order to save space and exploit signal naming convention.  
**MappingMetadata** class stores information about circuit topology, e.g., order and names of magnets in a particular circuit.  
There is a collection of csv files containing circuit topology summarised in the table below.

|System type|Hyperlink|
|------------|---------|
|beam mode   |https://gitlab.cern.ch/LHCData/lhc-sm-api/tree/master/lhcsmapi/metadata/beam_mode|
|busbar      |https://gitlab.cern.ch/LHCData/lhc-sm-api/tree/master/lhcsmapi/metadata/busbar|
|magnet      |https://gitlab.cern.ch/LHCData/lhc-sm-api/tree/master/lhcsmapi/metadata/magnet|
|qps_crate   |https://gitlab.cern.ch/LHCData/lhc-sm-api/tree/master/lhcsmapi/metadata/qps_crate|

- QPS crate name for a magnet 


```python
from lhcsmapi.metadata.MappingMetadata import MappingMetadata

MappingMetadata.get_crate_name_from_magnet_name('RB', 'MB.A16L2')
```




    'B16L2'



- QPS crates for RB


```python
from lhcsmapi.metadata.MappingMetadata import MappingMetadata

MappingMetadata.get_crates_for_circuit_names('RB', 'RB.A12')
```




    ['B8L2',
     'B10L2',
     'B12L2',
     'B14L2',
     'B16L2',
     'B18L2',
     'B20L2',
     'B22L2',
     'B24L2',
     'B26L2',
     'B28L2',
     'B30L2',
     'B32L2',
     'B34L2',
     'B33R1',
     'B31R1',
     'B29R1',
     'B27R1',
     'B25R1',
     'B23R1',
     'B21R1',
     'B19R1',
     'B17R1',
     'B15R1',
     'B13R1',
     'B11R1',
     'B9R1',
     'B8R1',
     'B10R1',
     'B12R1',
     'B14R1',
     'B16R1',
     'B18R1',
     'B20R1',
     'B22R1',
     'B24R1',
     'B26R1',
     'B28R1',
     'B30R1',
     'B32R1',
     'B34R1',
     'B33L2',
     'B31L2',
     'B29L2',
     'B27L2',
     'B25L2',
     'B23L2',
     'B21L2',
     'B19L2',
     'B17L2',
     'B15L2',
     'B13L2',
     'B11L2',
     'B9L2']



# 4. Reference
The Reference module stores information about reference signal profiles and features.

|System type|Hyperlink|
|-----------------|---------|
|Energy Extraction|https://gitlab.cern.ch/LHCData/lhc-sm-api/tree/master/lhcsmapi/reference/ee|
|Current Leads    |https://gitlab.cern.ch/LHCData/lhc-sm-api/tree/master/lhcsmapi/reference/leads|
|Power Converter  |https://gitlab.cern.ch/LHCData/lhc-sm-api/tree/master/lhcsmapi/reference/pc|
|Quench Heaters   |https://gitlab.cern.ch/LHCData/lhc-sm-api/tree/master/lhcsmapi/reference/qh|

References useful for analysis, so that we can measure deviation of signals.  
References change over time, so we need to keep track of the changes.

## 4.1. Get Reference Quench Heater Discharge Timestamp


```python
from lhcsmapi.reference.Reference import Reference

Reference.get_quench_heater_reference_discharge('RQ', '16L2')
```




    1417079205409000000



## 4.2. Get Reference Power Converter Timestamp


```python
from lhcsmapi.reference.Reference import Reference

Reference.get_power_converter_reference_fpa('RQ', 'RQD.A12', 'fgcPm')
```




    '2018-03-18 11:58:33.600'



## 4.3. Get Reference Energy Extraction Timestamp


```python
from lhcsmapi.reference.Reference import Reference

Reference.get_power_converter_reference_fpa('RQ', 'RQD.A12', 'eePm')
```




    '2018-03-18 11:58:33.675'



## 4.4. Get Reference Energy Extraction Parameters


```python
from lhcsmapi.reference.Reference import Reference

Reference.read_energy_extraction_reference_features('RB', '2018-03-18 11:58:33.675')
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>min</th>
      <th>max</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>R_1</th>
      <td>0.0675</td>
      <td>0.0825</td>
    </tr>
    <tr>
      <th>R_2</th>
      <td>0.0675</td>
      <td>0.0825</td>
    </tr>
    <tr>
      <th>t_delay_ee_1</th>
      <td>0.0500</td>
      <td>0.1500</td>
    </tr>
    <tr>
      <th>t_delay_ee_2</th>
      <td>0.5500</td>
      <td>0.6500</td>
    </tr>
    <tr>
      <th>tau_u_dump_res_1</th>
      <td>110.0000</td>
      <td>130.0000</td>
    </tr>
    <tr>
      <th>tau_u_dump_res_2</th>
      <td>110.0000</td>
      <td>130.0000</td>
    </tr>
  </tbody>
</table>
</div>




```python
from lhcsmapi.reference.Reference import Reference

Reference.read_energy_extraction_reference_features('RQ', '2018-03-18 11:58:33.675')
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>min</th>
      <th>max</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>t_delay_ee</th>
      <td>0.085</td>
      <td>0.115</td>
    </tr>
    <tr>
      <th>tau_u_dump_res</th>
      <td>25.000</td>
      <td>35.000</td>
    </tr>
  </tbody>
</table>
</div>



# 5. PmDbQuery
**PmDbRequest** class encapsulates PM database queries in order to simplify the query creation as well as browsing of PM events for a given period.

- Expected Success Response


```python
from lhcsmapi.dbsignal.post_mortem.PmDbRequest import PmDbRequest
source = 'RPTE.UA47.RB.A45'
system = 'FGC'
className = '51_self_pmd'
t_start = '2015-03-13 05:20:59.4910002'
duration = [(100, 's'), (100, 's')]
PmDbRequest.find_events(source=source, system=system, className=className, t_start=t_start, duration=duration)
```




    [('RPTE.UA47.RB.A45', 1426220469520000000)]



Source can be also replaced with a wildcard '*' in order to retrieve all sources for a given system, className, and duration.


```python
from lhcsmapi.dbsignal.post_mortem.PmDbRequest import PmDbRequest
source = "*"
system = "QPS"
className = "DQAMCNMB_PMHSU"
ts_start = 1426201200000000000
event_duration_in_sec = 24*60*60 # 1 day is the maximum duration for a single query

PmDbRequest.find_events(source, system, className, t_start=ts_start, duration=[(event_duration_in_sec, 's')])
```




    [('B20L5', 1426220469491000000),
     ('C20L5', 1426220517100000000),
     ('A20L5', 1426220518112000000),
     ('A21L5', 1426220625990000000),
     ('B21L5', 1426220866112000000),
     ('C23L4', 1426236802332000000),
     ('B23L4', 1426236839404000000),
     ('A23L4', 1426236839832000000),
     ('C22L4', 1426236949841000000),
     ('C15R4', 1426251285711000000),
     ('B15R4', 1426251337747000000),
     ('A15R4', 1426251388741000000),
     ('B34L8', 1426258716281000000),
     ('C34L8', 1426258747672000000),
     ('A34L8', 1426258747370000000),
     ('C33L8', 1426258835955000000),
     ('C34R7', 1426258853947000000),
     ('A34R7', 1426258854113000000),
     ('A20R3', 1426267931956000000),
     ('B20R3', 1426267983579000000),
     ('C20R3', 1426268004144000000),
     ('B18L5', 1426277626360000000),
     ('A18L5', 1426277679838000000),
     ('C18L5', 1426277680496000000),
     ('A19L5', 1426277903449000000)]



- Expected Response with no data or wrong input


```python
from lhcsmapi.dbsignal.post_mortem.PmDbRequest import PmDbRequest
source = "*"
system = "QPS"
className = "DQAMCNMB_PMHSU"
ts_start = 1426201200000000000
event_duration_in_sec = 1

PmDbRequest.find_events(source, system, className, t_start=ts_start, duration=[(event_duration_in_sec, 's')])
```




    []



# 6. DbSignal

We recommend the use of general purpose browsers for ad-hoc queries and exploration for signal hierarchy:
- The new PM web browser is available at the following link: http://post-mortem-paas.web.cern.ch/
- The new TIMBER NXCALS web browser is available at: https://timber.cern.ch/
- In order to find the mapping between CALS variables and NXCALS variable and metadata (along with the data migration status), please consult: https://ccde.cern.ch/dataBrowser/search?acwReportName=CALS-to-NXCALS%20Data%20Migration&domainName=CALS


In order to programatically query PM, CALS, or NXCALS database, a dedicated API along with parameters is required. An overview is summarised in the table below.

| &nbsp;          | PM                        | PM                                | CALS         | CALS*            | NXCALS                             | NXCALS                                       |
|-----------------|---------------------------|-----------------------------------|--------------|------------------|------------------------------------|----------------------------------------------|
| query type      | event                     | signal                            | signal       | feature          | signal                             | feature                                      |
| input           | system, source, className | system, source, className, signal | signal       | signal, features | system, (device, property), signal | system, (device, property), signal, features |
| time definition | time range                | timestamp                         | time range   | time range       | time range                         | time range                                   |
| time unit       | ns                        | ns                                | s            | s                | ns                                 | ns                                           |
| return type     | json                      | json                              | dict of list | dict of list     | spark dataframe                    | spark dataframe                              |
| execution time  | fast                      | fast                              | can be slow  | fast             | slow                               | fast                                         |
| execution type  | serial                    | serial                            | serial       | ?                | serial                             | parallel                                     |
| use             | simple                    | simple                            | simple       | simple           | simple                             | requires good knowledge of spark             |



The DbSignal provides a unified access to PM, CALS, and NXCALS databases in order to return a common data format, a pandas DataFrame. This enables a more advanced signal processing capabilities. The DbSignal module uses native API for each database:
- PM REST API: http://pm-api-pro/
- pytimber CALS API: https://github.com/rdemaria/pytimber
- NXCALS python API: http://nxcals-docs.web.cern.ch/0.4.11/


- PM signal query


```python
import requests
import json

response = requests.get('http://pm-api-pro/v2/pmdata/signal?system=FGC&className=51_self_pmd&source=RPTE.UA47.RB.A45&timestampInNanos=1426220469520000000&signal=STATUS.I_MEAS')
json_response = json.loads(response.text)

name = json_response['content'][0]['namesAndValues'][0]['name']
time = json_response['content'][0]['namesAndValues'][1]['value']
value = json_response['content'][0]['namesAndValues'][0]['value']

pd.DataFrame(value, time, [name]).plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f458c8fb630>



- CALS signal query


```python
import pandas as pd
import pytimber
ldb = pytimber.LoggingDB()

i_meas = ldb.get("RPTE.UA47.RB.A45:I_MEAS", '2015-03-13 05:20:49.4910002', '2015-03-13 05:22:49.4910002')
pd.DataFrame(i_meas.get("RPTE.UA47.RB.A45:I_MEAS")[1], i_meas.get("RPTE.UA47.RB.A45:I_MEAS")[0], ["RPTE.UA47.RB.A45:I_MEAS"]).plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f4568867fd0>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_84_1.png)


- NXCALS signal query


```python
from cern.nxcals.pyquery.builders import *
import pandas as pd

i_meas_df = DevicePropertyQuery.builder(spark).system('CMW') \
    .startTime(pd.Timestamp('2015-03-13T04:20:59.491000000').to_datetime64())\
    .endTime(pd.Timestamp('2015-03-13T04:22:49.491000000')) \
    .entity().device('RPTE.UA47.RB.A45').property('SUB') \
    .buildDataset().select('acqStamp', 'I_MEAS').dropna().sort("acqStamp").toPandas()

i_meas_df.set_index(i_meas_df['acqStamp'], inplace=True)
i_meas_df.drop(columns='acqStamp', inplace=True)
i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f45681447f0>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_86_1.png)



```python
from lhcsmapi.dbsignal.Signal import Signal

bmode_df = Signal().read('nxcals', nxcals_system='CMW',
              signal='HX:BMODE', t_start='2015-03-13 05:20:59.4910002', t_end='2015-04-13 05:20:59.4910002', spark=spark)
bmode_df.head()
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>HX:BMODE</th>
    </tr>
    <tr>
      <th>nxcals_timestamp</th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1426513864882000000</th>
      <td>SETUP</td>
    </tr>
    <tr>
      <th>1426515551191000000</th>
      <td>NOBEAM</td>
    </tr>
    <tr>
      <th>1426597426610000000</th>
      <td>SETUP</td>
    </tr>
    <tr>
      <th>1426597576404000000</th>
      <td>STABLE</td>
    </tr>
    <tr>
      <th>1426598591040000000</th>
      <td>UNSTABLE</td>
    </tr>
  </tbody>
</table>
</div>



Signal class encapsulates behavior for working with signal databases. It provides read and write methods for accessing Post Mortem, CALS and InfluxDB APIs.
<img src = "https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/signal.png" width=100%>

Read method supports only scalar arguments and returns a pandas Series.  
In case of errors, the read method returns a warning message with the raw API use to query in order to quickly reproduce the error. This way, one can exclude the error introduced by the lhcsmapi.

## 6.1. PM Database

- PM Success Response


```python
from lhcsmapi.dbsignal.Signal import Signal
system = 'FGC'
source = 'RPTE.UA47.RB.A45'
className = '51_self_pmd'
signal = 'STATUS.I_MEAS'
eventTime = 1426220469520000000
i_meas_df = Signal().read('pm', system=system, source=source, className=className, 
                          signal=signal, eventTime=eventTime)
i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f456813ad30>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_91_1.png)



```python
from lhcsmapi.dbsignal.Signal import Signal
system = "QPS"
className = "DQAMCNMB_PMHSU"
magnet = "B20L5"
signal = "I_HDS_4"
timestamp = 1426220469491000000
i_hds_4_df = Signal().read('pm', system=system, className=className, source=magnet,
                           signal=signal, eventTime=timestamp)
i_hds_4_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f456834c7f0>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_92_1.png)


- PM Fail Response


```python
from lhcsmapi.dbsignal.Signal import Signal
system = "QPS"
className = "DQAMCNMB_PMHSU"
magnet = "B20L5"
signal = "I_HDS_4"
timestamp = 142622046949100000 # last zero removed
i_hds_4_df = Signal().read('pm', system=system, className=className, source=magnet,
                              signal=signal, eventTime=timestamp)
```

    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/dbsignal/post_mortem/PmDbRequest.py:148: UserWarning: Querying Post Mortem failed using the following query: http://pm-api-pro/v2/pmdata/signal?system=QPS&className=DQAMCNMB_PMHSU&source=B20L5&timestampInNanos=142622046949100000&signal=I_HDS_4
      warnings.warn(warning)


## 6.2. CALS Database

- CALS Success Response

(t_start, t_end)


```python
from lhcsmapi.dbsignal.Signal import Signal
import pytimber
ldb = pytimber.LoggingDB()

signal = "RPTE.UA47.RB.A45:I_MEAS"
ts_start = 1426220459491000000
ts_end = 1426220479491000000
i_meas_df = Signal().read('cals', signal=signal, t_start=ts_start, t_end=ts_end, ldb=ldb)
i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f4552b5cda0>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_97_1.png)


(t_start, t_start+duration[0])


```python
from lhcsmapi.dbsignal.Signal import Signal
import pytimber
ldb = pytimber.LoggingDB()

signal = "RPTE.UA47.RB.A45:I_MEAS"
ts_start = 1426220459491000000
duration = [(20, 's')]
i_meas_df = Signal().read('cals', signal=signal, t_start=ts_start, duration=duration, ldb=ldb)
i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f4487655780>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_99_1.png)


(t_start-duration[0], t_start+duration[1])


```python
from lhcsmapi.dbsignal.Signal import Signal
import pytimber
ldb = pytimber.LoggingDB()

signal = "RPTE.UA47.RB.A45:I_MEAS"
ts_start = 1426220469491000000
duration = [(10, 's'), (10, 's')]
i_meas_df = Signal().read('cals', signal=signal, t_start=ts_start, duration=duration, ldb=ldb)
i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f44876351d0>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_101_1.png)


- CALS Fail Response


```python
from lhcsmapi.dbsignal.Signal import Signal
import pytimber
ldb = pytimber.LoggingDB()

signal = "RPTE.UA47.RB.A45:I_MEA" # last S missing
ts_start = 1426220469491000000
duration = [(10, 's'), (10, 's')]
i_meas_df = Signal().read('cals', signal=signal, t_start=ts_start, duration=duration, ldb=ldb)
```

    WARNING:pytimber.pytimber:No variables found.
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/dbsignal/cals/CalsDbSignal.py:179: UserWarning: Querying CALS with the parameters signal: RPTE.UA47.RB.A45:I_MEA, start time: 1426220459.4910002 and end time: 1426220479.4910002 returned no data.
                An empty DataFrame is returned.
                ___________________________________________________________________________________
                Reproduce the error using the raw pytimber API:
    
                import pytimber
                ldb = pytimber.LoggingDB()
                ldb.get("RPTE.UA47.RB.A45:I_MEA", 1426220459.4910002, 1426220479.4910002)
      warnings.warn(warning)


## 6.3. NXCALS Database

- NXCALS Success Response


```python
from lhcsmapi.dbsignal.Signal import Signal

nxcals_system = 'CMW'
nxcals_device = 'RPTE.UA47.RB.A45'
nxcals_property = 'SUB'
signal = 'I_MEAS'
t_start = 1426220469491000000
duration = [(10, 's'), (100, 's')]

i_meas_df = Signal().read('nxcals', nxcals_system=nxcals_system,  nxcals_device=nxcals_device, nxcals_property=nxcals_property, 
                          signal=signal, t_start=t_start, duration=duration, spark=spark)
i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f45681ac0f0>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_106_1.png)



```python
from lhcsmapi.dbsignal.Signal import Signal
nxcals_system = 'WINCCOA'

signal = 'DCBA.15R4.R:U_MAG'
t_start = 1426220469491000000
duration = [(10, 's'), (100, 's')]

u_mag_df = Signal().read('nxcals', nxcals_system=nxcals_system,
                          signal=signal, t_start=t_start, duration=duration, spark=spark)
u_mag_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f45680e17b8>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_107_1.png)


- NXCALS Fail Response


```python
from lhcsmapi.dbsignal.Signal import Signal
nxcals_system = 'CMW'
nxcals_device = 'RPTE.UA47.RB.A4' # missing last 5
nxcals_property = 'SUB'
signal = 'I_MEAS'
t_start = 1426220469491000000
duration = [(10, 's'), (100, 's')]

i_meas_df = Signal().read('nxcals', nxcals_system=nxcals_system,  nxcals_device=nxcals_device, nxcals_property=nxcals_property, 
                          signal=signal, t_start=t_start, duration=duration, spark=spark)
i_meas_df.plot()
```


    ---------------------------------------------------------------------------

    Py4JJavaError                             Traceback (most recent call last)

    /cvmfs/sft.cern.ch/lcg/views/LCG_95apython3_nxcals/x86_64-centos7-gcc7-opt/lib/python3.6/site-packages/pyspark/sql/utils.py in deco(*a, **kw)
         62         try:
    ---> 63             return f(*a, **kw)
         64         except py4j.protocol.Py4JJavaError as e:


    /cvmfs/sft.cern.ch/lcg/views/LCG_95apython3_nxcals/x86_64-centos7-gcc7-opt/lib/python3.6/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
        327                     "An error occurred while calling {0}{1}{2}.\n".
    --> 328                     format(target_id, ".", name), value)
        329             else:


    Py4JJavaError: An error occurred while calling o208.select.
    : org.apache.spark.sql.AnalysisException: cannot resolve '`acqStamp`' given input columns: [nxcals_entity_id, __record_version__, device, property, __record_timestamp__, class];;
    'Project ['acqStamp, 'I_MEAS]
    +- LocalRelation <empty>, [nxcals_entity_id#296L, device#297, property#298, class#299, __record_version__#300L, __record_timestamp__#301L]
    
    	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
    	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:110)
    	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:107)
    	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
    	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
    	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
    	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:277)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)
    	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:104)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:116)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$2.apply(QueryPlan.scala:121)
    	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
    	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:121)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2.apply(QueryPlan.scala:126)
    	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:126)
    	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:93)
    	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:107)
    	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)
    	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
    	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:85)
    	at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
    	at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
    	at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
    	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
    	at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
    	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
    	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
    	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
    	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
    	at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:3407)
    	at org.apache.spark.sql.Dataset.select(Dataset.scala:1335)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    	at py4j.Gateway.invoke(Gateway.java:282)
    	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    	at py4j.commands.CallCommand.execute(CallCommand.java:79)
    	at py4j.GatewayConnection.run(GatewayConnection.java:238)
    	at java.lang.Thread.run(Thread.java:745)


    
    During handling of the above exception, another exception occurred:


    AnalysisException                         Traceback (most recent call last)

    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/dbsignal/nxcals/NxcalsDbSignal.py in get_signal_df(signal, t_start, t_end, nxcals_system, nxcals_device, nxcals_property, spark)
         69                     .entity().device(nxcals_device).property(nxcals_property) \
    ---> 70                     .buildDataset().select('acqStamp', signal).dropna().sort("acqStamp").toPandas()
         71                 signal_df.set_index(signal_df['acqStamp'], inplace=True)


    /cvmfs/sft.cern.ch/lcg/views/LCG_95apython3_nxcals/x86_64-centos7-gcc7-opt/lib/python3.6/site-packages/pyspark/sql/dataframe.py in select(self, *cols)
       1319         """
    -> 1320         jdf = self._jdf.select(self._jcols(*cols))
       1321         return DataFrame(jdf, self.sql_ctx)


    /cvmfs/sft.cern.ch/lcg/views/LCG_95apython3_nxcals/x86_64-centos7-gcc7-opt/lib/python3.6/site-packages/py4j/java_gateway.py in __call__(self, *args)
       1256         return_value = get_return_value(
    -> 1257             answer, self.gateway_client, self.target_id, self.name)
       1258 


    /cvmfs/sft.cern.ch/lcg/views/LCG_95apython3_nxcals/x86_64-centos7-gcc7-opt/lib/python3.6/site-packages/pyspark/sql/utils.py in deco(*a, **kw)
         68             if s.startswith('org.apache.spark.sql.AnalysisException: '):
    ---> 69                 raise AnalysisException(s.split(': ', 1)[1], stackTrace)
         70             if s.startswith('org.apache.spark.sql.catalyst.analysis'):


    AnalysisException: "cannot resolve '`acqStamp`' given input columns: [nxcals_entity_id, __record_version__, device, property, __record_timestamp__, class];;\n'Project ['acqStamp, 'I_MEAS]\n+- LocalRelation <empty>, [nxcals_entity_id#296L, device#297, property#298, class#299, __record_version__#300L, __record_timestamp__#301L]\n"

    
    During handling of the above exception, another exception occurred:


    Exception                                 Traceback (most recent call last)

    <ipython-input-58-cfede295664b> in <module>
          8 
          9 i_meas_df = Signal().read('nxcals', nxcals_system=nxcals_system,  nxcals_device=nxcals_device, nxcals_property=nxcals_property, 
    ---> 10                           signal=signal, t_start=t_start, duration=duration, spark=spark)
         11 i_meas_df.plot()


    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/dbsignal/Signal.py in read(self, api, append, **kwargs)
        165         api = api.upper()
        166 
    --> 167         df = Signal._database_access_handler(api, 'read', **kwargs)
        168         if append:
        169             if self.__df_state is not None and self.__df_state != api:


    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/dbsignal/Signal.py in _database_access_handler(api, access_method_identifier, **kwargs)
        204         try:
        205             access_method = getattr(db_access, access_method_identifier, None)
    --> 206             return access_method(**kwargs)
        207         except AttributeError:
        208             raise AttributeError('{} Signal has no {} method'.format(api, access_method_identifier))


    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/dbsignal/nxcals/NxcalsDbSignal.py in read(self, signal, **kwargs)
         59         """
         60         return NxcalsDbSignal.get_signal_df(self.signal, self.t_start, self.t_end,
    ---> 61                                           self.nxcals_system, self.nxcals_device, self.nxcals_property, self.spark)
         62 
         63     @staticmethod


    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/dbsignal/nxcals/NxcalsDbSignal.py in get_signal_df(signal, t_start, t_end, nxcals_system, nxcals_device, nxcals_property, spark)
         80                             """.format(signal, t_start, t_end, nxcals_system, nxcals_device, nxcals_property)
         81 
    ---> 82                 raise Exception(exception)
         83         else:
         84             try:


    Exception: Querying NXCALS with the parameters signal: I_MEAS, start time: 2015-03-13T04:20:59.491000000, end time: 2015-03-13T04:22:49.491000000, nxcals_system CMW, nxcals_device RPTE.UA47.RB.A4, nxcals_property SUB failed
                                ___________________________________________________________________________________
                                Reproduce the error using the raw NXCALS API:
    
                                from cern.nxcals.pyquery.builders import *
                                DevicePropertyQuery.builder(spark).nxcals_system(''CMW'').startTime(''2015-03-13T04:20:59.491000000'').endTime(''2015-03-13T04:22:49.491000000'').entity().nxcals_device(''RPTE.UA47.RB.A4'').nxcals_property(''SUB'').buildDataset().select('acqStamp', ''signal'').dropna().sort("acqStamp").toPandas()
                                


# 7. pyeDSL - Embedded Domain Specific Language in python

The Signal class serves it purpose to get a raw data for a single signal. However, it makes it difficult to generalize in order to acquire the same signal across several circuits. In addition, typical processing requires to use additional classes and introduce even more local variables

```python
circuit_type = 'RB'
circuit_name = 'RB.A12'
t_start = '2015-01-13 16:59:11+01:00'
t_end = '2015-01-13 17:15:46+01:00'
db = 'NXCALS'
system = 'PC'

metadata_pc = SignalMetadata.get_circuit_signal_database_metadata(circuit_type, circuit_name, system, db)
I_MEAS = SignalMetadata.get_signal_name(circuit_type, circuit_name, system, db, 'I_MEAS')

i_meas_nxcals_df = Signal().read(db, signal=I_MEAS, t_start=t_start, t_end=t_end, 
                                 nxcals_device=metadata_pc['device'], nxcals_property=metadata_pc['property'], nxcals_system=metadata_pc['system'], 
                                 spark=spark)

i_meas_nxcals_df = SignalUtilities.synchronize_df(i_meas_nxcals_df)
i_meas_nxcals_df = SignalUtilities.convert_indices_to_sec(i_meas_nxcals_df)

```

Several design flaws leading to inconsistency and code duplications:
- <span style="color: red;">use of multiple methods, multiple arguments (duplicated across methods)</span>
- <span style="color: red;">multiple local variables (naming consistency across analysis modules)</span>
- <span style="color: red;">order of methods and arguments (with duck typing) not fixed</span>

The API also does not answer in a generic way the following questions
- What if we want to get current for each circuit?
- What if we want to get several current signals?

Natural languages have certain structure [1]

|Language|Word order|Example|
|--------|---------------------------|--------------|
|English:| {Subject}.{Verb}.{Object}:| John ate cake|
|Japanese:| {Subject}.{Order}.{Verb}:| John-ga keiki-o tabeta|
| | | 			John cake ate |
                
One can enforce syntactical order in code:
 - Domain Specific Language – new language, requires parser
 - Embedded Domain Specific Language – extends existing language
 
Furthermore, an eDSL could be implemented following the Fluent interface approach [2]. The use of an eDSL for signal query and processing is not a new concept as there exists already an eDSL in Java used to automatically check signals during Hardware Commisionning campaigns of the LHC [3].

[1] K. Gulordava, Word order variation and dependency length minimisation: a cross-linguistic computational approach, PhD thesis, UniGe,  
[2] https://en.wikipedia.org/wiki/Fluent_interface  
[3] M. Audrain, et al. - Using a Java Embedded Domain-Specific Language for LHC Test Analysis, ICALEPCS2013, San Francisco, CA, USA  


## 7.1. QueryBuilder()

We propose a python embedded Domain Specific Language (pyeDSL) for building queries:
- General purpose query
<center>{DB}.{DURATION}.{QUERY_PARAMETERS}.{QUERY}</center>

e.g.
```python
df = QueryBuilder().{with_pm()/with_cals()/with_nxcals()}.with_duration().with_query_parameters()\
	.signal_query().dfs[0]
df = QueryBuilder().{with_pm()/with_cals()/with_nxcals()}.with_timestamp().with_query_parameters()\
	.signal_query().dfs[0]
```

- Circuit-oriented query to generalize query across and within circuit types
<center>{DB}.{DURATION}.{CIRCUIT_TYPE}.{METADATA}.{QUERY}</center>

e.g.  

```python
df = QueryBuilder().{with_pm()/with_cals()/with_nxcals()}.with_duration().with_circuit_type().with_metadata()\
	.signal_query().dfs[0]
df = QueryBuilder().{with_pm()/with_cals()/with_nxcals()}.with_timestamp().with_circuit_type().with_metadata()\
	.signal_query().dfs[0]
```

+ <span style="color: green;">each parameter defined once (validation of input at each stage)</span>
+ <span style="color: green;">single local variable</span>
+ <span style="color: green;">order of operation is fixed</span>
+ <span style="color: green;">support for vector inputs</span>
+ <span style="color: green;">time-dependent metadata</span>


The pyeDSL provides hints on the order of execution


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

QueryBuilder()
```




    Set database name using with_pm(), with_cals(ldb), with_nxcals(spark) method.




```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

QueryBuilder().with_nxcals(spark)
```




    Database name properly set to NXCALS. Set time definition: for PM signal query, with_timestamp(),
     for PM event query or CALS, NXCALS signal query, with_duration()




```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

QueryBuilder().with_nxcals(spark).with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')])
```




    Query duration set to t_start=1426220359491000200, t_end=1426220559491000200. Set generic query parameter using with_query_parameters() method, or a circuit signal using with_circuit_type() method.



At the same time it prohibits unsupported operations throwing a meaningful exception


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

QueryBuilder().with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')])
```


    ---------------------------------------------------------------------------

    AttributeError                            Traceback (most recent call last)

    <ipython-input-62-17466de21b88> in <module>
          1 from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder
          2 
    ----> 3 QueryBuilder().with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')])
    

    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/QueryBuilder.py in __getattr__(self, name)
        132     def __getattr__(self, name):
        133         if name not in ['query', 'with_pm', 'with_cals', 'with_nxcals']:
    --> 134             raise AttributeError('{}() is not supported. {}'.format(name, self._msg))
        135 
        136     def __repr__(self):


    AttributeError: with_duration() is not supported. Set database name using with_pm(), with_cals(ldb), with_nxcals(spark) method.


Sections 7.1.1 - 7.1.6. deal with: PM event and signal query, CALS signal query, NXCALS signal and feature query (as shown in Table 1). Each case is discussed with a general purpose query, where the user has to provide full data on signal name and its metadata as well as circuit-oriented queries which provide a generic way of querying LHC circuit variables. Both query types are polymorphic and complemented with a set of post-processing functions.

*Since CALS is about to be discontinued at CERN, we do not support this type of query. For feature query with pytimber, please consult https://gitlab.cern.ch/scripting-tools/pytimber 

## 7.1.1. General-Purpose Query - Examples
A sentence constructed this way maintains the differences of query types while providing a common structure

- PM event query


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

source_timestamp_df = QueryBuilder().with_pm() \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')]) \
    .with_query_parameters(system='FGC', className='51_self_pmd', source='RPTE.UA47.RB.A45') \
    .event_query().df

source_timestamp_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>source</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>RPTE.UA47.RB.A45</td>
      <td>1426220469520000000</td>
    </tr>
  </tbody>
</table>
</div>




```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

source_timestamp_df = QueryBuilder().with_pm() \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(24*60*60, 's')]) \
    .with_query_parameters(system='QPS', className='DQAMCNMB_PMHSU', source='*') \
    .event_query().df

source_timestamp_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>source</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>B20L5</td>
      <td>1426220469491000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>C20L5</td>
      <td>1426220517100000000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A20L5</td>
      <td>1426220518112000000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A21L5</td>
      <td>1426220625990000000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>B21L5</td>
      <td>1426220866112000000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>C23L4</td>
      <td>1426236802332000000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>B23L4</td>
      <td>1426236839404000000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A23L4</td>
      <td>1426236839832000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>C22L4</td>
      <td>1426236949841000000</td>
    </tr>
    <tr>
      <th>9</th>
      <td>C15R4</td>
      <td>1426251285711000000</td>
    </tr>
    <tr>
      <th>10</th>
      <td>B15R4</td>
      <td>1426251337747000000</td>
    </tr>
    <tr>
      <th>11</th>
      <td>A15R4</td>
      <td>1426251388741000000</td>
    </tr>
    <tr>
      <th>12</th>
      <td>B34L8</td>
      <td>1426258716281000000</td>
    </tr>
    <tr>
      <th>13</th>
      <td>C34L8</td>
      <td>1426258747672000000</td>
    </tr>
    <tr>
      <th>14</th>
      <td>A34L8</td>
      <td>1426258747370000000</td>
    </tr>
    <tr>
      <th>15</th>
      <td>C33L8</td>
      <td>1426258835955000000</td>
    </tr>
    <tr>
      <th>16</th>
      <td>C34R7</td>
      <td>1426258853947000000</td>
    </tr>
    <tr>
      <th>17</th>
      <td>A34R7</td>
      <td>1426258854113000000</td>
    </tr>
    <tr>
      <th>18</th>
      <td>A20R3</td>
      <td>1426267931956000000</td>
    </tr>
    <tr>
      <th>19</th>
      <td>B20R3</td>
      <td>1426267983579000000</td>
    </tr>
    <tr>
      <th>20</th>
      <td>C20R3</td>
      <td>1426268004144000000</td>
    </tr>
    <tr>
      <th>21</th>
      <td>B18L5</td>
      <td>1426277626360000000</td>
    </tr>
    <tr>
      <th>22</th>
      <td>A18L5</td>
      <td>1426277679838000000</td>
    </tr>
    <tr>
      <th>23</th>
      <td>C18L5</td>
      <td>1426277680496000000</td>
    </tr>
    <tr>
      <th>24</th>
      <td>A19L5</td>
      <td>1426277903449000000</td>
    </tr>
  </tbody>
</table>
</div>



- PM signal query


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_pm() \
    .with_timestamp(1426220469520000000) \
    .with_query_parameters(system='FGC', source='RPTE.UA47.RB.A45', className='51_self_pmd', signal='STATUS.I_MEAS') \
    .signal_query().dfs[0]

i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f45680e6908>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_125_1.png)


- CALS signal query


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder
import pytimber
ldb = pytimber.LoggingDB()

i_meas_df = QueryBuilder().with_cals(ldb) \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')]) \
    .with_query_parameters(signal='RPTE.UA47.RB.A45:I_MEAS') \
    .signal_query().dfs[0]

i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f4487590048>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_127_1.png)


- NXCALS signal query - device, property


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')]) \
    .with_query_parameters(nxcals_system='CMW', nxcals_device='RPTE.UA47.RB.A45', nxcals_property='SUB', signal='I_MEAS') \
    .signal_query().dfs[0]

i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f455364a278>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_129_1.png)


- NXCALS signal query - variable


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

u_mag_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')]) \
    .with_query_parameters(nxcals_system='WINCCOA', signal='DCBA.15R4.R:U_MAG') \
    .signal_query().dfs[0]

u_mag_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f4553575e10>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_131_1.png)



```python
bmode_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', t_end='2015-04-13 05:20:59.4910002') \
    .with_query_parameters(nxcals_system='CMW', signal='HX:BMODE') \
    .signal_query().dfs[0]

bmode_df.head()
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>HX:BMODE</th>
    </tr>
    <tr>
      <th>nxcals_timestamp</th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1426513864882000000</th>
      <td>SETUP</td>
    </tr>
    <tr>
      <th>1426515551191000000</th>
      <td>NOBEAM</td>
    </tr>
    <tr>
      <th>1426597426610000000</th>
      <td>SETUP</td>
    </tr>
    <tr>
      <th>1426597576404000000</th>
      <td>STABLE</td>
    </tr>
    <tr>
      <th>1426598591040000000</th>
      <td>UNSTABLE</td>
    </tr>
  </tbody>
</table>
</div>



- NXCALS feature query - device, property

I_MEAS from beam injection, through beam acceleration, to stable beams


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=1526898157236000000, t_end=1526903352338000000) \
    .with_query_parameters(nxcals_system='CMW', nxcals_device='RPTE.UA23.RB.A12', nxcals_property='SUB', signal='I_MEAS') \
    .feature_query(['mean', 'std', 'max', 'min', 'count']).df

i_meas_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>device</th>
      <th>max</th>
      <th>min</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>RPTE.UA23.RB.A12</td>
      <td>10978.8</td>
      <td>757.18</td>
      <td>2498</td>
      <td>5374.768659</td>
      <td>3315.972856</td>
    </tr>
  </tbody>
</table>
</div>



- NXCALS feature query - variable

U_MAG from beam injection, through beam acceleration, to stable beams


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

u_mag_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=1526898157236000000, t_end=1526903352338000000) \
    .with_query_parameters(nxcals_system='WINCCOA', signal='DCBB.8L2.R:U_MAG') \
    .feature_query(['mean', 'std', 'max', 'min', 'count']).df

u_mag_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>nxcals_variable_name</th>
      <th>max</th>
      <th>min</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>0.002113</td>
      <td>-0.975332</td>
      <td>51951</td>
      <td>-0.196706</td>
      <td>0.375195</td>
    </tr>
  </tbody>
</table>
</div>



## 7.1.2. General-Purpose Query - Polymorphism
- multiple signal, mutliple sources, multiple systems, multiple className


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_pm() \
    .with_timestamp(1426220469520000000) \
    .with_query_parameters(system=['FGC', 'FGC'], source=['RPTE.UA47.RB.A45', 'RPTE.UA47.RB.A45'], 
                           className=['51_self_pmd', '51_self_pmd'], signal=['STATUS.I_MEAS', 'STATUS.I_REF']) \
    .signal_query().dfs

ax=i_meas_df[0].plot()
i_meas_df[1].plot(ax=ax)
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f45534e1e10>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_138_1.png)


- multiple signal, single source, single system, single className


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_pm() \
    .with_timestamp(1426220469520000000) \
    .with_query_parameters(system='FGC', source='RPTE.UA47.RB.A45', 
                           className='51_self_pmd', signal=['STATUS.I_MEAS', 'STATUS.I_REF']) \
    .signal_query().dfs

ax=i_meas_df[0].plot()
i_meas_df[1].plot(ax=ax)
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f45533e2780>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_140_1.png)


## 7.1.3. Circuit-Oriented Query - Examples
- PM event query


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

source_timestamp_df = QueryBuilder().with_pm() \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='PC') \
    .event_query().df

source_timestamp_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>source</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>RPTE.UA47.RB.A45</td>
      <td>1426220469520000000</td>
    </tr>
  </tbody>
</table>
</div>




```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

source_timestamp_df = QueryBuilder().with_pm() \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(24*60*60, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='QH', source='*') \
    .event_query().df

source_timestamp_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>source</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>B20L5</td>
      <td>1426220469491000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>C20L5</td>
      <td>1426220517100000000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A20L5</td>
      <td>1426220518112000000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A21L5</td>
      <td>1426220625990000000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>B21L5</td>
      <td>1426220866112000000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>C23L4</td>
      <td>1426236802332000000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>B23L4</td>
      <td>1426236839404000000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A23L4</td>
      <td>1426236839832000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>C22L4</td>
      <td>1426236949841000000</td>
    </tr>
    <tr>
      <th>9</th>
      <td>C15R4</td>
      <td>1426251285711000000</td>
    </tr>
    <tr>
      <th>10</th>
      <td>B15R4</td>
      <td>1426251337747000000</td>
    </tr>
    <tr>
      <th>11</th>
      <td>A15R4</td>
      <td>1426251388741000000</td>
    </tr>
    <tr>
      <th>12</th>
      <td>B34L8</td>
      <td>1426258716281000000</td>
    </tr>
    <tr>
      <th>13</th>
      <td>C34L8</td>
      <td>1426258747672000000</td>
    </tr>
    <tr>
      <th>14</th>
      <td>A34L8</td>
      <td>1426258747370000000</td>
    </tr>
    <tr>
      <th>15</th>
      <td>C33L8</td>
      <td>1426258835955000000</td>
    </tr>
    <tr>
      <th>16</th>
      <td>C34R7</td>
      <td>1426258853947000000</td>
    </tr>
    <tr>
      <th>17</th>
      <td>A34R7</td>
      <td>1426258854113000000</td>
    </tr>
    <tr>
      <th>18</th>
      <td>A20R3</td>
      <td>1426267931956000000</td>
    </tr>
    <tr>
      <th>19</th>
      <td>B20R3</td>
      <td>1426267983579000000</td>
    </tr>
    <tr>
      <th>20</th>
      <td>C20R3</td>
      <td>1426268004144000000</td>
    </tr>
    <tr>
      <th>21</th>
      <td>B18L5</td>
      <td>1426277626360000000</td>
    </tr>
    <tr>
      <th>22</th>
      <td>A18L5</td>
      <td>1426277679838000000</td>
    </tr>
    <tr>
      <th>23</th>
      <td>C18L5</td>
      <td>1426277680496000000</td>
    </tr>
    <tr>
      <th>24</th>
      <td>A19L5</td>
      <td>1426277903449000000</td>
    </tr>
  </tbody>
</table>
</div>



- PM signal query


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_pm() \
    .with_timestamp(1426220469520000000) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='PC', signal='I_MEAS') \
    .signal_query().dfs[0]

i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f4553398208>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_145_1.png)


- CALS signal query


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder
import pytimber
ldb = pytimber.LoggingDB()

i_meas_df = QueryBuilder().with_cals(ldb) \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='PC', signal='I_MEAS')\
    .signal_query().dfs[0]

i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f4487557908>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_147_1.png)


- NXCALS signal query - device, property


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='PC', signal='I_MEAS') \
    .signal_query().dfs[0]

i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f455336bd68>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_149_1.png)


- NXCALS signal query - variable


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

u_mag_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='BUSBAR', signal='U_MAG', wildcard={'BUSBAR': 'DCBA.15R4.R'}) \
    .signal_query().dfs[0]

u_mag_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f4553273b70>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_151_1.png)


- NXCALS feature query - device, property

I_MEAS from beam injection, through beam acceleration, to stable beams


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=1526898157236000000, t_end=1526903352338000000) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system='PC', signal='I_MEAS') \
    .feature_query(['mean', 'std', 'max', 'min', 'count']).df

i_meas_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>device</th>
      <th>max</th>
      <th>min</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>RPTE.UA23.RB.A12</td>
      <td>10978.8</td>
      <td>757.18</td>
      <td>2498</td>
      <td>5374.768659</td>
      <td>3315.972856</td>
    </tr>
  </tbody>
</table>
</div>



- NXCALS feature query - variable

U_MAG from beam injection, through beam acceleration, to stable beams


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

u_mag_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=1526898157236000000, t_end=1526903352338000000) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system='BUSBAR', signal='U_MAG', wildcard={'BUSBAR': 'DCBB.8L2.R'}) \
    .feature_query(['mean', 'std', 'max', 'min', 'count']).df

u_mag_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>nxcals_variable_name</th>
      <th>max</th>
      <th>min</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>0.002113</td>
      <td>-0.975332</td>
      <td>51951</td>
      <td>-0.196706</td>
      <td>0.375195</td>
    </tr>
  </tbody>
</table>
</div>



## 7.1.4. Circuit-Oriented Query - Polymorphism
- Multiple circuit names


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_dfs = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(100, 's'), (100, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name=['RB.A12', 'RB.A45'], system='PC', signal='I_MEAS')\
    .signal_query().dfs

ax = i_meas_dfs[0].plot()
i_meas_dfs[1].plot(ax=ax, grid=True)
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f45531eff98>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_157_1.png)


- Multiple system names


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

u_hts_dfs = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=1544622149598000000, duration=[(50, 's'), (150, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system=['LEADS_EVEN', 'LEADS_ODD'], signal='U_HTS') \
    .signal_query().dfs

ax = u_hts_dfs[0].plot()
u_hts_dfs[1].plot(ax=ax)
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f4553148358>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_159_1.png)


- Multiple signal names


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

u_ext_dfs = QueryBuilder().with_pm() \
    .with_timestamp(1544622149598000000) \
    .with_circuit_type('RQ') \
    .with_metadata(circuit_name='RQD.A12', system='QDS', signal=['U_1_EXT', 'U_2_EXT'],
                        source='16L2', wildcard={'CELL': '16L2'}) \
    .signal_query().dfs

ax = u_ext_dfs[0].plot()
u_ext_dfs[1].plot(ax=ax)
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f45533a7710>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_161_1.png)


- Signal wildcard


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder
import matplotlib.pyplot as plt
import pytimber
ldb = pytimber.LoggingDB()

u_diode_rqd_dfs = QueryBuilder().with_cals(ldb) \
    .with_duration(t_start=int(1544622149613000000), duration=[(50, 's'), (150, 's')]) \
    .with_circuit_type('RQ') \
    .with_metadata(circuit_name='RQD.A12', system='DIODE_RQD', signal='U_DIODE_RQD', wildcard={'MAGNET': '*'})\
    .signal_query().dfs

# plot all
fig, ax = plt.subplots()
for u_diode_rqd_df in u_diode_rqd_dfs:
    u_diode_rqd_df.plot(ax=ax)
ax.legend().set_visible(False)
```


![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_163_0.png)


## 7.1.5. Advanced Feature Query

NXCALS enables calculation of signal features such as min, max, mean, std, count directly on the cluster without the need for costly query of the signal and performing calculation locally. This approach enables parallel computing on the cluster. To this end, a query should contain an element enabling a group by operation. Each group by operation allows for executing computation in parallel. For the sake of compactness, we only show examples for circuit-oriented query, however, the same principle applies to the general-purpose queries.
- Feature query of multiple signals for the same period of time


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=1526898157236000000, t_end=1526903352338000000) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='*', system='PC', signal='I_MEAS') \
    .feature_query(['mean', 'std', 'max', 'min', 'count']).df

i_meas_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>device</th>
      <th>max</th>
      <th>min</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>RPTE.UA63.RB.A56</td>
      <td>10974.36</td>
      <td>756.90</td>
      <td>2498</td>
      <td>5372.636369</td>
      <td>3314.646638</td>
    </tr>
    <tr>
      <th>1</th>
      <td>RPTE.UA87.RB.A81</td>
      <td>10973.21</td>
      <td>756.82</td>
      <td>2498</td>
      <td>5371.596990</td>
      <td>3314.230379</td>
    </tr>
    <tr>
      <th>2</th>
      <td>RPTE.UA83.RB.A78</td>
      <td>10967.94</td>
      <td>756.47</td>
      <td>2498</td>
      <td>5369.226934</td>
      <td>3312.676833</td>
    </tr>
    <tr>
      <th>3</th>
      <td>RPTE.UA27.RB.A23</td>
      <td>10976.46</td>
      <td>757.04</td>
      <td>2498</td>
      <td>5373.973151</td>
      <td>3315.324716</td>
    </tr>
    <tr>
      <th>4</th>
      <td>RPTE.UA43.RB.A34</td>
      <td>10975.16</td>
      <td>756.95</td>
      <td>2497</td>
      <td>5373.559131</td>
      <td>3314.074552</td>
    </tr>
    <tr>
      <th>5</th>
      <td>RPTE.UA67.RB.A67</td>
      <td>10978.56</td>
      <td>757.18</td>
      <td>2498</td>
      <td>5374.329179</td>
      <td>3315.855302</td>
    </tr>
    <tr>
      <th>6</th>
      <td>RPTE.UA47.RB.A45</td>
      <td>10971.44</td>
      <td>756.71</td>
      <td>2497</td>
      <td>5370.954557</td>
      <td>3312.837184</td>
    </tr>
    <tr>
      <th>7</th>
      <td>RPTE.UA23.RB.A12</td>
      <td>10978.80</td>
      <td>757.18</td>
      <td>2498</td>
      <td>5374.768659</td>
      <td>3315.972856</td>
    </tr>
  </tbody>
</table>
</div>



- Feature query of multiple signals with the same period of time subdivided into three intervals - group by signal name and interval


```python
t_start_injs = 1526898157236000000
t_end_injs = 1526899957236000000
t_start_sbs = 1526901552338000000
t_end_sbs = 1526903352338000000
```


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=t_start_injs, t_end=t_end_sbs) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system='PC', signal='I_MEAS')\
    .signal_query().dfs[0]

ax = i_meas_df.plot(figsize=(10,5), linewidth=5)
ax.axvspan(xmin=t_start_injs, xmax=t_end_injs, facecolor='xkcd:goldenrod')
ax.axvspan(xmin=t_end_injs, xmax=t_start_sbs, facecolor='xkcd:grey')
ax.axvspan(xmin=t_start_sbs, xmax=t_end_sbs, facecolor='xkcd:green')
```




    <matplotlib.patches.Polygon at 0x7f45526d3518>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_168_1.png)


Function translate introduces a mapping based on the time column. Here, we consider three subintervals for beam injection, beam acceleration, and stable beams.

As a result, the time column forms a partition and can be executed in parallel.


```python
from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

def translate(timestamp):
    if(timestamp >= t_start_injs and timestamp < t_end_injs):
        return 1
    
    if(timestamp >=  t_end_injs and timestamp < t_start_sbs):
        return 2
    
    if(timestamp >= t_start_sbs and timestamp <= t_end_sbs):
        return 3
    
    return -1

translate_udf = udf(translate, IntegerType())
```

The translate function should be passed as a function argument


```python
i_meas_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=t_start_injs, t_end=t_end_sbs) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='*', system='PC', signal='I_MEAS') \
    .feature_query(['mean', 'std', 'max', 'min', 'count'], function=translate_udf).df

i_meas_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>device</th>
      <th>class</th>
      <th>max</th>
      <th>min</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>RPTE.UA43.RB.A34</td>
      <td>1</td>
      <td>756.96</td>
      <td>756.95</td>
      <td>60</td>
      <td>756.959833</td>
      <td>0.001291</td>
    </tr>
    <tr>
      <th>1</th>
      <td>RPTE.UA63.RB.A56</td>
      <td>3</td>
      <td>10974.36</td>
      <td>10974.35</td>
      <td>60</td>
      <td>10974.350167</td>
      <td>0.001291</td>
    </tr>
    <tr>
      <th>2</th>
      <td>RPTE.UA47.RB.A45</td>
      <td>3</td>
      <td>10971.44</td>
      <td>10971.43</td>
      <td>60</td>
      <td>10971.439667</td>
      <td>0.001810</td>
    </tr>
    <tr>
      <th>3</th>
      <td>RPTE.UA47.RB.A45</td>
      <td>2</td>
      <td>10970.89</td>
      <td>756.71</td>
      <td>2377</td>
      <td>5346.059971</td>
      <td>3193.562789</td>
    </tr>
    <tr>
      <th>4</th>
      <td>RPTE.UA23.RB.A12</td>
      <td>3</td>
      <td>10978.80</td>
      <td>10978.77</td>
      <td>60</td>
      <td>10978.781500</td>
      <td>0.007552</td>
    </tr>
    <tr>
      <th>5</th>
      <td>RPTE.UA83.RB.A78</td>
      <td>2</td>
      <td>10967.54</td>
      <td>756.47</td>
      <td>2378</td>
      <td>5344.349941</td>
      <td>3193.572017</td>
    </tr>
    <tr>
      <th>6</th>
      <td>RPTE.UA43.RB.A34</td>
      <td>2</td>
      <td>10974.68</td>
      <td>756.95</td>
      <td>2377</td>
      <td>5348.695949</td>
      <td>3194.776339</td>
    </tr>
    <tr>
      <th>7</th>
      <td>RPTE.UA67.RB.A67</td>
      <td>1</td>
      <td>757.19</td>
      <td>757.18</td>
      <td>60</td>
      <td>757.188833</td>
      <td>0.003237</td>
    </tr>
    <tr>
      <th>8</th>
      <td>RPTE.UA87.RB.A81</td>
      <td>3</td>
      <td>10973.21</td>
      <td>10973.20</td>
      <td>60</td>
      <td>10973.209167</td>
      <td>0.002787</td>
    </tr>
    <tr>
      <th>9</th>
      <td>RPTE.UA27.RB.A23</td>
      <td>2</td>
      <td>10976.10</td>
      <td>757.04</td>
      <td>2378</td>
      <td>5349.106434</td>
      <td>3196.138736</td>
    </tr>
    <tr>
      <th>10</th>
      <td>RPTE.UA83.RB.A78</td>
      <td>3</td>
      <td>10967.94</td>
      <td>10967.94</td>
      <td>60</td>
      <td>10967.940000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>11</th>
      <td>RPTE.UA47.RB.A45</td>
      <td>1</td>
      <td>756.71</td>
      <td>756.71</td>
      <td>60</td>
      <td>756.710000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>12</th>
      <td>RPTE.UA83.RB.A78</td>
      <td>1</td>
      <td>756.48</td>
      <td>756.47</td>
      <td>60</td>
      <td>756.472000</td>
      <td>0.004034</td>
    </tr>
    <tr>
      <th>13</th>
      <td>RPTE.UA27.RB.A23</td>
      <td>1</td>
      <td>757.05</td>
      <td>757.04</td>
      <td>60</td>
      <td>757.046833</td>
      <td>0.004691</td>
    </tr>
    <tr>
      <th>14</th>
      <td>RPTE.UA87.RB.A81</td>
      <td>2</td>
      <td>10972.79</td>
      <td>756.83</td>
      <td>2378</td>
      <td>5346.697637</td>
      <td>3195.063002</td>
    </tr>
    <tr>
      <th>15</th>
      <td>RPTE.UA87.RB.A81</td>
      <td>1</td>
      <td>756.83</td>
      <td>756.82</td>
      <td>60</td>
      <td>756.829167</td>
      <td>0.002787</td>
    </tr>
    <tr>
      <th>16</th>
      <td>RPTE.UA23.RB.A12</td>
      <td>2</td>
      <td>10978.41</td>
      <td>757.19</td>
      <td>2378</td>
      <td>5349.879361</td>
      <td>3196.753501</td>
    </tr>
    <tr>
      <th>17</th>
      <td>RPTE.UA67.RB.A67</td>
      <td>2</td>
      <td>10978.15</td>
      <td>757.18</td>
      <td>2378</td>
      <td>5349.423865</td>
      <td>3196.631627</td>
    </tr>
    <tr>
      <th>18</th>
      <td>RPTE.UA63.RB.A56</td>
      <td>2</td>
      <td>10973.97</td>
      <td>756.90</td>
      <td>2378</td>
      <td>5347.758797</td>
      <td>3195.477077</td>
    </tr>
    <tr>
      <th>19</th>
      <td>RPTE.UA43.RB.A34</td>
      <td>3</td>
      <td>10975.16</td>
      <td>10975.15</td>
      <td>60</td>
      <td>10975.154833</td>
      <td>0.005039</td>
    </tr>
    <tr>
      <th>20</th>
      <td>RPTE.UA23.RB.A12</td>
      <td>1</td>
      <td>757.22</td>
      <td>757.18</td>
      <td>60</td>
      <td>757.201667</td>
      <td>0.007847</td>
    </tr>
    <tr>
      <th>21</th>
      <td>RPTE.UA27.RB.A23</td>
      <td>3</td>
      <td>10976.46</td>
      <td>10976.45</td>
      <td>60</td>
      <td>10976.450333</td>
      <td>0.001810</td>
    </tr>
    <tr>
      <th>22</th>
      <td>RPTE.UA63.RB.A56</td>
      <td>1</td>
      <td>756.91</td>
      <td>756.90</td>
      <td>60</td>
      <td>756.903667</td>
      <td>0.004860</td>
    </tr>
    <tr>
      <th>23</th>
      <td>RPTE.UA67.RB.A67</td>
      <td>3</td>
      <td>10978.56</td>
      <td>10978.55</td>
      <td>60</td>
      <td>10978.550167</td>
      <td>0.001291</td>
    </tr>
  </tbody>
</table>
</div>



The same method applied to NXCALS signals based on variable


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

u_mag_ab_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=t_start_injs, t_end=t_end_sbs) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system='BUSBAR', signal='U_MAG', wildcard={'BUSBAR': 'DCBB.8L2.R'}) \
    .feature_query(['mean', 'std', 'max', 'min', 'count'], function=translate_udf).df

u_mag_ab_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>nxcals_variable_name</th>
      <th>class</th>
      <th>max</th>
      <th>min</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>3</td>
      <td>0.002003</td>
      <td>-0.034035</td>
      <td>18000</td>
      <td>-0.001983</td>
      <td>0.002882</td>
    </tr>
    <tr>
      <th>1</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>1</td>
      <td>0.002113</td>
      <td>-0.005992</td>
      <td>18000</td>
      <td>-0.001861</td>
      <td>0.002844</td>
    </tr>
    <tr>
      <th>2</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>2</td>
      <td>0.002041</td>
      <td>-0.975332</td>
      <td>15951</td>
      <td>-0.636315</td>
      <td>0.423767</td>
    </tr>
  </tbody>
</table>
</div>



This method can be used together with signal decimation, i.e., taking every nth sample.

For example this can be useful to query QPS board A and B which share the same channel and samples are shifted by 5 so that 
- every 10-th sample belongs to board A (or B), decimation=10


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

u_mag_a_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=t_start_injs, t_end=t_end_sbs) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system='BUSBAR', signal='U_MAG', wildcard={'BUSBAR': 'DCBB.8L2.R'}) \
    .feature_query(['mean', 'std', 'max', 'min', 'count'], function=translate_udf, decimation=10).df

u_mag_a_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>nxcals_variable_name</th>
      <th>class</th>
      <th>max</th>
      <th>min</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>3</td>
      <td>0.002003</td>
      <td>-0.012705</td>
      <td>1800</td>
      <td>0.000812</td>
      <td>0.000529</td>
    </tr>
    <tr>
      <th>1</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>1</td>
      <td>0.002113</td>
      <td>-0.000463</td>
      <td>1800</td>
      <td>0.000965</td>
      <td>0.000434</td>
    </tr>
    <tr>
      <th>2</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>2</td>
      <td>0.002041</td>
      <td>-0.969405</td>
      <td>1595</td>
      <td>-0.633601</td>
      <td>0.423847</td>
    </tr>
  </tbody>
</table>
</div>



- every 5+10-th sample belongs to board B (or A), decimation=10, shift=5


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

u_mag_b_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=t_start_injs, t_end=t_end_sbs) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system='BUSBAR', signal='U_MAG', wildcard={'BUSBAR': 'DCBB.8L2.R'}) \
    .feature_query(['mean', 'std', 'max', 'min', 'count'], function=translate_udf, decimation=10, shift=5).df

u_mag_b_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>nxcals_variable_name</th>
      <th>class</th>
      <th>max</th>
      <th>min</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>3</td>
      <td>-0.003425</td>
      <td>-0.034035</td>
      <td>1800</td>
      <td>-0.004754</td>
      <td>0.000877</td>
    </tr>
    <tr>
      <th>1</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>1</td>
      <td>-0.003361</td>
      <td>-0.005992</td>
      <td>1800</td>
      <td>-0.004656</td>
      <td>0.000441</td>
    </tr>
    <tr>
      <th>2</th>
      <td>DCBB.8L2.R:U_MAG</td>
      <td>2</td>
      <td>-0.003052</td>
      <td>-0.975332</td>
      <td>1595</td>
      <td>-0.639233</td>
      <td>0.423941</td>
    </tr>
  </tbody>
</table>
</div>



- with polymorphism one can query 1248 busbar at once (in two batches of 624 due to the limit of 1000 signal per query)


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

u_mag_ab_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=t_start_injs, t_end=t_end_sbs) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='*', system='BUSBAR', signal='U_MAG', wildcard={'BUSBAR': '*'}) \
    .feature_query(['mean', 'std', 'max', 'min', 'count'], function=translate_udf).df

u_mag_ab_df.head()
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>nxcals_variable_name</th>
      <th>class</th>
      <th>max</th>
      <th>min</th>
      <th>count</th>
      <th>mean</th>
      <th>std</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>DCBA.22L4.L:U_MAG</td>
      <td>1</td>
      <td>-0.000167</td>
      <td>-0.002102</td>
      <td>18000</td>
      <td>-0.001110</td>
      <td>0.000639</td>
    </tr>
    <tr>
      <th>1</th>
      <td>DCBA.14L4.R:U_MAG</td>
      <td>1</td>
      <td>0.001179</td>
      <td>-0.000796</td>
      <td>18000</td>
      <td>0.000171</td>
      <td>0.000603</td>
    </tr>
    <tr>
      <th>2</th>
      <td>DCBA.A28R2.L:U_MAG</td>
      <td>3</td>
      <td>0.045531</td>
      <td>-0.003455</td>
      <td>18000</td>
      <td>-0.002131</td>
      <td>0.001361</td>
    </tr>
    <tr>
      <th>3</th>
      <td>DCBB.B22L2.R:U_MAG</td>
      <td>1</td>
      <td>0.000341</td>
      <td>-0.003088</td>
      <td>18000</td>
      <td>-0.001283</td>
      <td>0.000557</td>
    </tr>
    <tr>
      <th>4</th>
      <td>DCBB.11L2.R:U_MAG</td>
      <td>3</td>
      <td>0.048651</td>
      <td>-0.002121</td>
      <td>18000</td>
      <td>0.001457</td>
      <td>0.002587</td>
    </tr>
  </tbody>
</table>
</div>



## 7.1.6. Processing Raw Signals
Once a signal is queried, one can perform some operations on each of them.  
In this case, the order of operations does not matter (but can be checked).  

|Signal query|Signal processing|
| ---------- | :-------------- |
|{DB}.{DURATION}.{QUERY_PARAMETERS}.{QUERY}| |
|{DB}.{DURATION}.{CIRCUIT_TYPE}.{METADATA}.{QUERY}| |
|&nbsp;|.synchronize_time()|
|&nbsp;|.convert_index_to_sec()|
|&nbsp;|.create_col_from_index()|
|&nbsp;|.filter_median()|
|&nbsp;|.remove_values_for_time_less_than()|
|&nbsp;|.remove_initial_offset()|


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

i_meas_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start='2015-01-13 16:59:11+01:00', t_end='2015-01-13 17:15:46+01:00') \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system='PC', signal='I_MEAS') \
    .signal_query() \
    .synchronize_time() \
    .convert_index_to_sec().dfs[0]

i_meas_df.plot()
```




    <matplotlib.axes._subplots.AxesSubplot at 0x7f45528afda0>




![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_182_1.png)


## 7.1.7. Processing Raw Events
For PM event queries one can perform several operations on source, timestamp dataframe. 

|Event query|Event processing|
| ---------- | :-------------- |
|{DB}.{DURATION}.{QUERY_PARAMETERS}.{QUERY}| |
|{DB}.{DURATION}.{CIRCUIT_TYPE}.{METADATA}.{QUERY}| |
|&nbsp;|.filter_source()|
|&nbsp;|.drop_duplicate_source()|
|&nbsp;|.sort_values()|

The processing methods are dedicated to performing repeated operations on PM events. In case of searching a given system and className with wildcard '\*' as a source, the event query can return events from different sectors. In this case, one can filter events to contain to a given sector. Some PM systems return duplicate events from different types of boards. In this case one can drop duplicate sources. Eventually, the events can be sorted by either source or timestamp.


- Filter source


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

source_timestamp_df = QueryBuilder().with_pm() \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(24*60*60, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='QH', source='*') \
    .event_query() \
    .df

source_timestamp_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>source</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>B20L5</td>
      <td>1426220469491000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>C20L5</td>
      <td>1426220517100000000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A20L5</td>
      <td>1426220518112000000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A21L5</td>
      <td>1426220625990000000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>B21L5</td>
      <td>1426220866112000000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>C23L4</td>
      <td>1426236802332000000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>B23L4</td>
      <td>1426236839404000000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A23L4</td>
      <td>1426236839832000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>C22L4</td>
      <td>1426236949841000000</td>
    </tr>
    <tr>
      <th>9</th>
      <td>C15R4</td>
      <td>1426251285711000000</td>
    </tr>
    <tr>
      <th>10</th>
      <td>B15R4</td>
      <td>1426251337747000000</td>
    </tr>
    <tr>
      <th>11</th>
      <td>A15R4</td>
      <td>1426251388741000000</td>
    </tr>
    <tr>
      <th>12</th>
      <td>B34L8</td>
      <td>1426258716281000000</td>
    </tr>
    <tr>
      <th>13</th>
      <td>C34L8</td>
      <td>1426258747672000000</td>
    </tr>
    <tr>
      <th>14</th>
      <td>A34L8</td>
      <td>1426258747370000000</td>
    </tr>
    <tr>
      <th>15</th>
      <td>C33L8</td>
      <td>1426258835955000000</td>
    </tr>
    <tr>
      <th>16</th>
      <td>C34R7</td>
      <td>1426258853947000000</td>
    </tr>
    <tr>
      <th>17</th>
      <td>A34R7</td>
      <td>1426258854113000000</td>
    </tr>
    <tr>
      <th>18</th>
      <td>A20R3</td>
      <td>1426267931956000000</td>
    </tr>
    <tr>
      <th>19</th>
      <td>B20R3</td>
      <td>1426267983579000000</td>
    </tr>
    <tr>
      <th>20</th>
      <td>C20R3</td>
      <td>1426268004144000000</td>
    </tr>
    <tr>
      <th>21</th>
      <td>B18L5</td>
      <td>1426277626360000000</td>
    </tr>
    <tr>
      <th>22</th>
      <td>A18L5</td>
      <td>1426277679838000000</td>
    </tr>
    <tr>
      <th>23</th>
      <td>C18L5</td>
      <td>1426277680496000000</td>
    </tr>
    <tr>
      <th>24</th>
      <td>A19L5</td>
      <td>1426277903449000000</td>
    </tr>
  </tbody>
</table>
</div>



Executing filter_source() with circuit name and system type would filter out events not belonging to a given circuit name


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

source_timestamp_df = QueryBuilder().with_pm() \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(24*60*60, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='QH', source='*') \
    .event_query() \
    .filter_source('RB.A45', 'QH') \
    .df

source_timestamp_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>source</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>B20L5</td>
      <td>1426220469491000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>C20L5</td>
      <td>1426220517100000000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A20L5</td>
      <td>1426220518112000000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A21L5</td>
      <td>1426220625990000000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>B21L5</td>
      <td>1426220866112000000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>C15R4</td>
      <td>1426251285711000000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>B15R4</td>
      <td>1426251337747000000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A15R4</td>
      <td>1426251388741000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>B18L5</td>
      <td>1426277626360000000</td>
    </tr>
    <tr>
      <th>9</th>
      <td>A18L5</td>
      <td>1426277679838000000</td>
    </tr>
    <tr>
      <th>10</th>
      <td>C18L5</td>
      <td>1426277680496000000</td>
    </tr>
    <tr>
      <th>11</th>
      <td>A19L5</td>
      <td>1426277903449000000</td>
    </tr>
  </tbody>
</table>
</div>



- Drop duplicates


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

source_timestamp_df = QueryBuilder().with_pm() \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(24*60*60, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='QDS', source='*') \
    .event_query() \
    .df

source_timestamp_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>source</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>B20L5</td>
      <td>1426220469490000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>C20L5</td>
      <td>1426220517099000000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A20L5</td>
      <td>1426220518111000000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A21L5</td>
      <td>1426220625989000000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>B21L5</td>
      <td>1426220866111000000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>B20L5</td>
      <td>1426220469492000000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>C20L5</td>
      <td>1426220517101000000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A20L5</td>
      <td>1426220518113000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>A21L5</td>
      <td>1426220625991000000</td>
    </tr>
    <tr>
      <th>9</th>
      <td>B21L5</td>
      <td>1426220866113000000</td>
    </tr>
    <tr>
      <th>10</th>
      <td>C23L4</td>
      <td>1426236802331000000</td>
    </tr>
    <tr>
      <th>11</th>
      <td>B23L4</td>
      <td>1426236839403000000</td>
    </tr>
    <tr>
      <th>12</th>
      <td>A23L4</td>
      <td>1426236839831000000</td>
    </tr>
    <tr>
      <th>13</th>
      <td>C22L4</td>
      <td>1426236949840000000</td>
    </tr>
    <tr>
      <th>14</th>
      <td>C23L4</td>
      <td>1426236802333000000</td>
    </tr>
    <tr>
      <th>15</th>
      <td>B23L4</td>
      <td>1426236839405000000</td>
    </tr>
    <tr>
      <th>16</th>
      <td>A23L4</td>
      <td>1426236839833000000</td>
    </tr>
    <tr>
      <th>17</th>
      <td>C22L4</td>
      <td>1426236949842000000</td>
    </tr>
    <tr>
      <th>18</th>
      <td>C15R4</td>
      <td>1426251285710000000</td>
    </tr>
    <tr>
      <th>19</th>
      <td>B15R4</td>
      <td>1426251337746000000</td>
    </tr>
    <tr>
      <th>20</th>
      <td>A15R4</td>
      <td>1426251388740000000</td>
    </tr>
    <tr>
      <th>21</th>
      <td>C15R4</td>
      <td>1426251285712000000</td>
    </tr>
    <tr>
      <th>22</th>
      <td>B15R4</td>
      <td>1426251337748000000</td>
    </tr>
    <tr>
      <th>23</th>
      <td>A15R4</td>
      <td>1426251388742000000</td>
    </tr>
    <tr>
      <th>24</th>
      <td>B34L8</td>
      <td>1426258716280000000</td>
    </tr>
    <tr>
      <th>25</th>
      <td>C34L8</td>
      <td>1426258747671000000</td>
    </tr>
    <tr>
      <th>26</th>
      <td>A34L8</td>
      <td>1426258747369000000</td>
    </tr>
    <tr>
      <th>27</th>
      <td>C33L8</td>
      <td>1426258835954000000</td>
    </tr>
    <tr>
      <th>28</th>
      <td>C34R7</td>
      <td>1426258853946000000</td>
    </tr>
    <tr>
      <th>29</th>
      <td>A34R7</td>
      <td>1426258854112000000</td>
    </tr>
    <tr>
      <th>30</th>
      <td>B34L8</td>
      <td>1426258716282000000</td>
    </tr>
    <tr>
      <th>31</th>
      <td>C34L8</td>
      <td>1426258747673000000</td>
    </tr>
    <tr>
      <th>32</th>
      <td>A34L8</td>
      <td>1426258747371000000</td>
    </tr>
    <tr>
      <th>33</th>
      <td>C33L8</td>
      <td>1426258835956000000</td>
    </tr>
    <tr>
      <th>34</th>
      <td>C34R7</td>
      <td>1426258853948000000</td>
    </tr>
    <tr>
      <th>35</th>
      <td>A34R7</td>
      <td>1426258854114000000</td>
    </tr>
    <tr>
      <th>36</th>
      <td>A20R3</td>
      <td>1426267931955000000</td>
    </tr>
    <tr>
      <th>37</th>
      <td>B20R3</td>
      <td>1426267983578000000</td>
    </tr>
    <tr>
      <th>38</th>
      <td>C20R3</td>
      <td>1426268004143000000</td>
    </tr>
    <tr>
      <th>39</th>
      <td>A20R3</td>
      <td>1426267931957000000</td>
    </tr>
    <tr>
      <th>40</th>
      <td>B20R3</td>
      <td>1426267983580000000</td>
    </tr>
    <tr>
      <th>41</th>
      <td>C20R3</td>
      <td>1426268004145000000</td>
    </tr>
    <tr>
      <th>42</th>
      <td>B18L5</td>
      <td>1426277626359000000</td>
    </tr>
    <tr>
      <th>43</th>
      <td>A18L5</td>
      <td>1426277679837000000</td>
    </tr>
    <tr>
      <th>44</th>
      <td>C18L5</td>
      <td>1426277680495000000</td>
    </tr>
    <tr>
      <th>45</th>
      <td>A19L5</td>
      <td>1426277903448000000</td>
    </tr>
    <tr>
      <th>46</th>
      <td>B18L5</td>
      <td>1426277626361000000</td>
    </tr>
    <tr>
      <th>47</th>
      <td>A18L5</td>
      <td>1426277679839000000</td>
    </tr>
    <tr>
      <th>48</th>
      <td>C18L5</td>
      <td>1426277680497000000</td>
    </tr>
    <tr>
      <th>49</th>
      <td>A19L5</td>
      <td>1426277903450000000</td>
    </tr>
  </tbody>
</table>
</div>



Executing filter_source() with circuit name and system type would filter out events not belonging to a given circuit name


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

source_timestamp_df = QueryBuilder().with_pm() \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(24*60*60, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='QDS', source='*') \
    .event_query() \
    .filter_source('RB.A45', 'QDS') \
    .df

source_timestamp_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>source</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>B20L5</td>
      <td>1426220469490000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>C20L5</td>
      <td>1426220517099000000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A20L5</td>
      <td>1426220518111000000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A21L5</td>
      <td>1426220625989000000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>B21L5</td>
      <td>1426220866111000000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>B20L5</td>
      <td>1426220469492000000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>C20L5</td>
      <td>1426220517101000000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A20L5</td>
      <td>1426220518113000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>A21L5</td>
      <td>1426220625991000000</td>
    </tr>
    <tr>
      <th>9</th>
      <td>B21L5</td>
      <td>1426220866113000000</td>
    </tr>
    <tr>
      <th>10</th>
      <td>C15R4</td>
      <td>1426251285710000000</td>
    </tr>
    <tr>
      <th>11</th>
      <td>B15R4</td>
      <td>1426251337746000000</td>
    </tr>
    <tr>
      <th>12</th>
      <td>A15R4</td>
      <td>1426251388740000000</td>
    </tr>
    <tr>
      <th>13</th>
      <td>C15R4</td>
      <td>1426251285712000000</td>
    </tr>
    <tr>
      <th>14</th>
      <td>B15R4</td>
      <td>1426251337748000000</td>
    </tr>
    <tr>
      <th>15</th>
      <td>A15R4</td>
      <td>1426251388742000000</td>
    </tr>
    <tr>
      <th>16</th>
      <td>B18L5</td>
      <td>1426277626359000000</td>
    </tr>
    <tr>
      <th>17</th>
      <td>A18L5</td>
      <td>1426277679837000000</td>
    </tr>
    <tr>
      <th>18</th>
      <td>C18L5</td>
      <td>1426277680495000000</td>
    </tr>
    <tr>
      <th>19</th>
      <td>A19L5</td>
      <td>1426277903448000000</td>
    </tr>
    <tr>
      <th>20</th>
      <td>B18L5</td>
      <td>1426277626361000000</td>
    </tr>
    <tr>
      <th>21</th>
      <td>A18L5</td>
      <td>1426277679839000000</td>
    </tr>
    <tr>
      <th>22</th>
      <td>C18L5</td>
      <td>1426277680497000000</td>
    </tr>
    <tr>
      <th>23</th>
      <td>A19L5</td>
      <td>1426277903450000000</td>
    </tr>
  </tbody>
</table>
</div>



- drop_duplicate_source()

Some PM systems return duplicate events from different types of boards. In this case one can drop duplicate sources. 


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

source_timestamp_df = QueryBuilder().with_pm() \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(24*60*60, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='QDS', source='*') \
    .event_query() \
    .filter_source('RB.A45', 'QDS') \
    .drop_duplicate_source() \
    .df

source_timestamp_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>source</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>B20L5</td>
      <td>1426220469490000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>C20L5</td>
      <td>1426220517099000000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A20L5</td>
      <td>1426220518111000000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A21L5</td>
      <td>1426220625989000000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>B21L5</td>
      <td>1426220866111000000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>C15R4</td>
      <td>1426251285710000000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>B15R4</td>
      <td>1426251337746000000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A15R4</td>
      <td>1426251388740000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>B18L5</td>
      <td>1426277626359000000</td>
    </tr>
    <tr>
      <th>9</th>
      <td>A18L5</td>
      <td>1426277679837000000</td>
    </tr>
    <tr>
      <th>10</th>
      <td>C18L5</td>
      <td>1426277680495000000</td>
    </tr>
    <tr>
      <th>11</th>
      <td>A19L5</td>
      <td>1426277903448000000</td>
    </tr>
  </tbody>
</table>
</div>



- sort_values()

The events can be sorted by either source or timestamp.


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder

source_timestamp_df = QueryBuilder().with_pm() \
    .with_duration(t_start='2015-03-13 05:20:59.4910002', duration=[(24*60*60, 's')]) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A45', system='QDS', source='*') \
    .event_query() \
    .filter_source('RB.A45', 'QDS') \
    .drop_duplicate_source() \
    .sort_values(by='timestamp') \
    .df

source_timestamp_df
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>source</th>
      <th>timestamp</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>B20L5</td>
      <td>1426220469490000000</td>
    </tr>
    <tr>
      <th>1</th>
      <td>C20L5</td>
      <td>1426220517099000000</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A20L5</td>
      <td>1426220518111000000</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A21L5</td>
      <td>1426220625989000000</td>
    </tr>
    <tr>
      <th>4</th>
      <td>B21L5</td>
      <td>1426220866111000000</td>
    </tr>
    <tr>
      <th>5</th>
      <td>C15R4</td>
      <td>1426251285710000000</td>
    </tr>
    <tr>
      <th>6</th>
      <td>B15R4</td>
      <td>1426251337746000000</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A15R4</td>
      <td>1426251388740000000</td>
    </tr>
    <tr>
      <th>8</th>
      <td>B18L5</td>
      <td>1426277626359000000</td>
    </tr>
    <tr>
      <th>9</th>
      <td>A18L5</td>
      <td>1426277679837000000</td>
    </tr>
    <tr>
      <th>10</th>
      <td>C18L5</td>
      <td>1426277680495000000</td>
    </tr>
    <tr>
      <th>11</th>
      <td>A19L5</td>
      <td>1426277903448000000</td>
    </tr>
  </tbody>
</table>
</div>



## 7.2. AssertionBuilder()

<center>{SIGNALS}.(TIME_RANGE).{ASSERTION}

|Signal input|Time range definition (optional) / Signal assertion|Signal assertions (if time range defined)|
| ---------- | :------------------ |-----------------|
|.with_signal()|&nbsp;|&nbsp;|
|&nbsp;|.has_min_max_value()|&nbsp;|
|&nbsp;|.compare_to_reference()|&nbsp;|
|&nbsp;|.with_time_range()|.has_min_max_variation()|
|&nbsp;|.with_time_range()|.has_min_max_slope()|

- has_min_max_value()
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/MP3.png">


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder
from lhcsmapi.pyedsl.AssertionBuilder import AssertionBuilder

tt891a_dfs = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start='2014-12-13 09:12:41+01:00', t_end='2014-12-13 12:27:11+01:00') \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system=['LEADS_EVEN', 'LEADS_ODD'], signal='TT891A') \
    .signal_query() \
    .synchronize_time() \
    .convert_index_to_sec() \
    .filter_median().dfs


AssertionBuilder().with_signal(tt891a_dfs) \
    .has_min_max_value(value_min=46, value_max=54)
```


![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_198_0.png)


In the case below, the assertion should fail and raise a warning.


```python
AssertionBuilder().with_signal(tt891a_dfs) \
    .has_min_max_value(value_min=50, value_max=54)
```

    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:63: UserWarning: DACA05_07L2_TT891A.TEMPERATURECALC outside of the [50, 54] mV threshold
      warnings.warn('{} outside of the [{}, {}] mV threshold'.format(col, value_min, value_max))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:63: UserWarning: DACA06_07L2_TT891A.TEMPERATURECALC outside of the [50, 54] mV threshold
      warnings.warn('{} outside of the [{}, {}] mV threshold'.format(col, value_min, value_max))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:63: UserWarning: DABA01_07R1_TT891A.TEMPERATURECALC outside of the [50, 54] mV threshold
      warnings.warn('{} outside of the [{}, {}] mV threshold'.format(col, value_min, value_max))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:63: UserWarning: DABA02_07R1_TT891A.TEMPERATURECALC outside of the [50, 54] mV threshold
      warnings.warn('{} outside of the [{}, {}] mV threshold'.format(col, value_min, value_max))



![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_200_1.png)


- compare_to_reference()


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder
from lhcsmapi.pyedsl.AssertionBuilder import AssertionBuilder
from lhcsmapi.reference.Reference import Reference
from lhcsmapi.Time import Time

timestamp_ee_rqd = 1544622149701000000
timestamp_fgc_rqd = 1544622149620000000
signal_names = 'T_RES'


t_res_df = QueryBuilder().with_pm() \
    .with_timestamp(timestamp_ee_rqd) \
    .with_circuit_type('RQ') \
    .with_metadata(circuit_name='RQD.A12', system='EE', signal=signal_names).signal_query() \
    .remove_values_for_time_less_than(timestamp_ee_rqd) \
    .synchronize_time(timestamp_fgc_rqd) \
    .convert_index_to_sec().dfs[0]


timestamp_ee_ref_rqd = Reference.get_power_converter_reference_fpa('RQ', 'RQD.A12', 'eePm')
timestamp_ee_ref_rqd = Time.to_unix_timestamp(timestamp_ee_ref_rqd)

timestamp_fgc_ref_rqd = Reference.get_power_converter_reference_fpa('RQ', 'RQD.A12', 'fgcPm')
timestamp_fgc_ref_rqd = Time.to_unix_timestamp(timestamp_fgc_ref_rqd)


t_res_ref_df = QueryBuilder().with_pm() \
    .with_timestamp(timestamp_ee_ref_rqd) \
    .with_circuit_type('RQ') \
    .with_metadata(circuit_name='RQD.A12', system='EE', signal=signal_names).signal_query() \
    .remove_values_for_time_less_than(timestamp_ee_ref_rqd) \
    .synchronize_time(timestamp_fgc_ref_rqd) \
    .convert_index_to_sec().dfs[0]

AssertionBuilder().with_signal([t_res_df])\
    .compare_to_reference(signal_ref_dfs=[t_res_ref_df], abs_margin=25, scaling=1)
```


![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_202_0.png)


- has_min_max_variation()


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder
from lhcsmapi.pyedsl.AssertionBuilder import AssertionBuilder
from lhcsmapi.analysis.busbar.BusbarResistanceAnalysis import BusbarResistanceAnalysis

t_start = '2014-12-13 09:12:41+01:00'
t_end = '2014-12-13 12:27:11+01:00'

cv891_dfs = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=t_start, t_end=t_end) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system=['LEADS_EVEN', 'LEADS_ODD'], signal='CV891') \
    .signal_query() \
    .synchronize_time() \
    .convert_index_to_sec() \
    .filter_median() \
    .dfs

i_meas_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=t_start, t_end=t_end) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system='PC', signal='I_MEAS') \
    .signal_query() \
    .create_col_from_index('acqStamp') \
    .synchronize_time() \
    .convert_index_to_sec() \
    .dfs[0]

plateau_start, plateau_end = BusbarResistanceAnalysis.get_current_plateau_start_end(i_meas_df, 'I_MEAS', i_meas_threshold=500)
plateau_start_sync = [(ps-i_meas_df['acqStamp'].values[0])/1e9 for ps in plateau_start]
plateau_end_sync = [(pe-i_meas_df['acqStamp'].values[0])/1e9 for pe in plateau_end]

AssertionBuilder().with_signal(cv891_dfs) \
    .with_time_range(t_start=plateau_start_sync, t_end=plateau_end_sync) \
    .has_min_max_variation(variation_min_max=8)
```


![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_204_0.png)


In the case below, the variation is too tight and the assertion fails 


```python
AssertionBuilder().with_signal(cv891_dfs) \
    .with_time_range(t_start=plateau_start_sync, t_end=plateau_end_sync) \
    .has_min_max_variation(variation_min_max=1)
```

    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:134: UserWarning: The variation of DACA05_07L2_CV891.POSST (1.0 %) exceeds 1.5 % for constant current from 806.0 to 5882.5 s
      .format(col, variation_min_max, variation.values[0], t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:134: UserWarning: The variation of DACA06_07L2_CV891.POSST (1.0 %) exceeds 1.9000000000000021 % for constant current from 806.0 to 5882.5 s
      .format(col, variation_min_max, variation.values[0], t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:134: UserWarning: The variation of DABA01_07R1_CV891.POSST (1.0 %) exceeds 1.6999999999999993 % for constant current from 806.0 to 5882.5 s
      .format(col, variation_min_max, variation.values[0], t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:134: UserWarning: The variation of DABA01_07R1_CV891.POSST (1.0 %) exceeds 1.1000000000000014 % for constant current from 6199.0 to 9745.0 s
      .format(col, variation_min_max, variation.values[0], t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:134: UserWarning: The variation of DABA02_07R1_CV891.POSST (1.0 %) exceeds 2.099999999999998 % for constant current from 806.0 to 5882.5 s
      .format(col, variation_min_max, variation.values[0], t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:134: UserWarning: The variation of DABA02_07R1_CV891.POSST (1.0 %) exceeds 1.2000000000000028 % for constant current from 6199.0 to 9745.0 s
      .format(col, variation_min_max, variation.values[0], t_s, t_e))



![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_206_1.png)


- has_min_max_slope()


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder
from lhcsmapi.pyedsl.AssertionBuilder import AssertionBuilder
from lhcsmapi.analysis.busbar.BusbarResistanceAnalysis import BusbarResistanceAnalysis

t_start = '2014-12-13 09:12:41+01:00'
t_end = '2014-12-13 12:27:11+01:00'

u_res_dfs = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=t_start, t_end=t_end) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system=['LEADS_EVEN', 'LEADS_ODD'], signal='U_RES') \
    .signal_query() \
    .synchronize_time() \
    .convert_index_to_sec() \
    .filter_median() \
    .dfs

i_meas_df = QueryBuilder().with_nxcals(spark) \
    .with_duration(t_start=t_start, t_end=t_end) \
    .with_circuit_type('RB') \
    .with_metadata(circuit_name='RB.A12', system='PC', signal='I_MEAS') \
    .signal_query() \
    .create_col_from_index('acqStamp') \
    .synchronize_time() \
    .convert_index_to_sec() \
    .dfs[0]

plateau_start, plateau_end = BusbarResistanceAnalysis.get_current_plateau_start_end(i_meas_df, 'I_MEAS', i_meas_threshold=500)
plateau_start_sync = [(ps-i_meas_df['acqStamp'].values[0])/1e9 for ps in plateau_start]
plateau_end_sync = [(pe-i_meas_df['acqStamp'].values[0])/1e9 for pe in plateau_end]

AssertionBuilder().with_signal(u_res_dfs) \
    .with_time_range(t_start=plateau_start_sync, t_end=plateau_end_sync) \
    .has_min_max_slope(slope_min=-2, slope_max=2, ylabel='U_RES')
```


![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_208_0.png)


In the case below, the slope is too tight and the assertion fails 


```python
AssertionBuilder().with_signal(u_res_dfs) \
    .with_time_range(t_start=plateau_start_sync, t_end=plateau_end_sync) \
    .has_min_max_slope(slope_min=-2e-3, slope_max=2e-3, ylabel='U_RES')
```

    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:152: UserWarning: The drift of DFLAS.7L2.RB.A12.LD1:U_RES is -0.119 mV/h for constant current from 806.0 to 5882.5 s
      .format(col, slope, unit, t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:152: UserWarning: The drift of DFLAS.7L2.RB.A12.LD1:U_RES is -0.667 mV/h for constant current from 6199.0 to 9745.0 s
      .format(col, slope, unit, t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:152: UserWarning: The drift of DFLAS.7L2.RB.A12.LD2:U_RES is 0.159 mV/h for constant current from 806.0 to 5882.5 s
      .format(col, slope, unit, t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:152: UserWarning: The drift of DFLAS.7L2.RB.A12.LD2:U_RES is 0.636 mV/h for constant current from 6199.0 to 9745.0 s
      .format(col, slope, unit, t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:152: UserWarning: The drift of DFLAS.7R1.RB.A12.LD3:U_RES is 0.205 mV/h for constant current from 806.0 to 5882.5 s
      .format(col, slope, unit, t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:152: UserWarning: The drift of DFLAS.7R1.RB.A12.LD3:U_RES is 0.421 mV/h for constant current from 6199.0 to 9745.0 s
      .format(col, slope, unit, t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:152: UserWarning: The drift of DFLAS.7R1.RB.A12.LD4:U_RES is -0.188 mV/h for constant current from 806.0 to 5882.5 s
      .format(col, slope, unit, t_s, t_e))
    /eos/home-m/mmacieje/SWAN_projects/lhc-sm-api/lhcsmapi/pyedsl/AssertionBuilder.py:152: UserWarning: The drift of DFLAS.7R1.RB.A12.LD4:U_RES is -0.469 mV/h for constant current from 6199.0 to 9745.0 s
      .format(col, slope, unit, t_s, t_e))



![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_210_1.png)


## 7.3. FeatureBuilder()

<center>{SIGNALS}.(FEATURE_CALCULATION).{ASSERTION}</center>

e.g.  
```python
FeatureBuilder().with_signal(u_hds_dfs) \
                .calculate_features(features=['first', 'last20mean', 'tau_charge'], index=1544622149599000000)
```
Supported functions are: 

```python
['first', 'first20mean', 'last', 'last20mean', 'max', 'min', 'median', 'std', 'mean', 'tau_charge', 'tau_energy', 'tau_lin_reg', 'tau_exp_fit']
```
For example, to calculate initial voltage, final mean voltage based on the last 20 points, and the characteristic time of the pseudo-exponential decay for a quench heater voltage.
<img src = "https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/QH.png">


```python
from lhcsmapi.pyedsl.QueryBuilder import QueryBuilder
from lhcsmapi.pyedsl.FeatureBuilder import FeatureBuilder
import matplotlib.pyplot as plt

timestamp = 1544622149599000000

u_hds_dfs = QueryBuilder().with_pm() \
    .with_timestamp(timestamp) \
    .with_circuit_type('RQ') \
    .with_metadata(circuit_name='RQD.A12', system='QH', signal='U_HDS', source='16L2', wildcard={'CELL': '16L2'}) \
    .signal_query()\
    .synchronize_time(timestamp)\
    .convert_index_to_sec().dfs

ax = u_hds_dfs[0].plot(figsize=(15,7))
u_hds_dfs[1].plot(ax=ax, grid=True)
plt.show()

FeatureBuilder().with_signal(u_hds_dfs) \
                .calculate_features(features=['first', 'last20mean', 'tau_charge'], index=1544622149599000000)
```


![png](https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/output/output_212_0.png)





<div>

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>16L2:U_HDS_1:first</th>
      <th>16L2:U_HDS_1:last20mean</th>
      <th>16L2:U_HDS_1:tau_charge</th>
      <th>16L2:U_HDS_2:first</th>
      <th>16L2:U_HDS_2:last20mean</th>
      <th>16L2:U_HDS_2:tau_charge</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1544622149599000000</th>
      <td>880.4621</td>
      <td>5.575086</td>
      <td>0.07779</td>
      <td>872.8354</td>
      <td>6.734703</td>
      <td>0.076704</td>
    </tr>
  </tbody>
</table>
</div>



# Known Issues
In order to update the lhcsmapi package one has to execute the command
```
pip install --user --update lhcsmapi
```
twice while using the SWAN terminal (cf. an error message in the figure below).
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-api/raw/master/figures/double_reinstallation_error.png">
