Jscatter’s documentation¶
The aim of Jscatter is treatment of experimental data and models:

Reading and analyzing experimental data with associated attributes as temperature, wavevector, comment, ….
Multidimensional fitting taking attributes into account.
Providing useful models for neutron and X-ray scattering form factors, structure factors and dynamic models (quasi elastic neutron scattering) and other topics.
Simplified plotting with paper ready quality.
Easy model building for non programmers.
Python scripts/Jupyter Notebooks to document data evaluation and modelling.
Main concept
Link data from experiment, analytical model or simulation with attributes as .temperature, .wavevector, .pressure,…
Methods for fitting, filter, merging,… using the attributes by name.
Provide an extensible library with common theories/models for fitting of physical models.
Allowing evaluation even of hundreds of datasets using scripts.
Data organisation
Multiple measurements are stored in a
dataList
(subclass of list) containingdataArray
(subclass of numpy ndarray) for each measurement. Both allow attributes to contain additional information of the measurement and have special attributes as .X,.Y,.eY,…- for convenience and easy reading. See What are dataArray/dataList.
Read/Write data
The intention is to read everything (with comments) from an ASCII file to use it later if needed. Multiple measurement files can be read at once and then filtered according to attributes to get subsets. See Reading ASCII files.
Fitting
Multidimensional, attribute dependent fitting (least square Levenberg-Marquardt, differential evolution, …from scipy.optimize). Attributes are used automatically as fixed fit parameters. See Fitting experimental data,
fit()
, 1D fits with attributes.
Plotting
The aim is to provide one line plotting commands to allow a fast view on data, with the possibility to pretty up the plots.
We use an adaption of Xmgrace for 2D plots (a wrapper; see GracePlot) as it allows interactive publication ready output in high quality for 2D plots and is much faster than matplotlib.
A small matplotlib interface mpl is provided and matplotlib can be used as it is (e.g. for 3D plots).
Still any other plotting package can be used.
Model Library
By intention the user should write own models or modify existing ones to combine different contributions (to include e.g. a background, instrument resolution, …).
Models can be defined as normal functions within a script or in interactive session of (I)python. See How to build simple models and How to build a more complex model .
The model library contains general purpose routines e.g. for vectorized quadrature (formel) or specialised models for scattering in formfactor (ff), structurefactor (sf) and dynamic. Models save parameters as attributes for later access/documentation. The model library can also be used for other purposes and may be extended by users need.
Some special functions:
scatteringLengthDensityCalc()
-> Electron density, coh and inc neutron scattering length, masswaterdensity()
-> Density of water (H2O/D2O) with inorganic substancessedimentationProfile()
-> The Lamm equation of sedimenting particlesRMSA()
-> Rescaled MSA structure factor for dilute charged colloidal dispersionshydrodynamicFunct()
-> Hydrodynamic function from hydrodynamic pair interactionmultiShellSphere()
-> Formfactor of multi shell spherical particlesmultiShellCylinder()
-> Formfactor of multi shell cylinder particles with capsorientedCloudScattering()
-> 2D scattering of an oriented cloud of scatterersfiniteZimm()
-> Zimm model with internal friction -> intermediate scattering functiondiffusionHarmonicPotential()
-> Diffusion in harmonic potential-> intermediate scattering functionsmear()
-> Smearing enabling simultaneous fits of differently smeared SANS/SAXS datadesmear()
-> Desmearing according to the Lake algorithm for the abovewaterXrayScattering()
-> Absolute scattering of water with components (salt, buffer)
- How to use Jscatter
see Examples and Beginners Guide / Help or try Jscatter live at
.
# import jscatter and numpy
import numpy as np
import jscatter as js
# read the data (16 dataArrays) with attributes as q, Dtrans .... into dataList
i5 = js.dL(js.examples.datapath + '/iqt_1hho.dat')
# define a model for the fit
diffusion = lambda A, D, t, wavevector, e=0: A*np.exp(-wavevector**2*D*t) + e
# do the fit
# single valued start parameters are the same for all 16 dataArrays
# list start parameters [...] indicate independent fitting for dataArrays
# the command line shows progress and the final result, which is found in i5.lastfit
i5.fit(model=diffusion, # the fit function
freepar={'D': [0.08], 'A': 0.98}, # free start parameters
fixpar={'e': 0.0}, # fixed parameters
mapNames={'t': 'X', 'wavevector': 'q'}) # map names from model to data
# open plot with results and residuals
i5.showlastErrPlot(yscale='l')
# open a plot with fixed size and plot data and fit result
p = js.grace(1.2, 0.8)
# plot the data with Q values in legend as symbols
p.plot(i5, symbol=[-1, 0.4, -1], legend='Q=$q')
# plot fit results in lastfit as lines without symbol or legend
p.plot(i5.lastfit, symbol=0, line=[1, 1, -1])
# pretty up if needed
p.yaxis(min=0.02, max=1.1, scale='log', charsize=1.5, label='I(Q,t)/I(Q,0)')
p.xaxis(min=0, max=130, charsize=1.5, label='t / ns')
p.legend(x=110, y=0.9, charsize=1)
p.title('I(Q,t) as measured by Neutron Spinecho Spectroscopy', size=1.3)
p.text('for diffusion a single exp. decay', x=60, y=0.35, rot=360 - 20, color=4)
p.text(r'f(t)=A*e\S-Q\S2\N\SDt', x=100, y=0.025, rot=0, charsize=1.5)
if 1: # optional; save in different formats
p.save('DiffusionFit.agr')
p.save('DiffusionFit.png')

Shortcuts:
import jscatter as js
js.showDoc() # Show html documentation in browser
exampledA=js.dA('test.dat') # shortcut to create dataArray from file
exampledL=js.dL('test.dat') # shortcut to create dataList from file
p=js.grace() # create plot in XmGrace
p=js.mplot() # create plot in matplotlib
p.plot(exampledL) # plot the read dataList
Jscatter package contents¶
- 1. Beginners Guide / Help
- 1.1. What are dataArray/dataList
- 1.2. First basic examples
- 1.3. Reading ASCII files
- 1.4. Creating from numpy arrays
- 1.5. Manipulating dataArray/dataList
- 1.6. Indexing dataArray/dataList and reducing
- 1.7. Fitting experimental data
- 1.8. Why Fits may fail
- 1.9. Plot experimental data and fit result
- 1.10. Save data and fit results
- 1.11. Additional stuff
- 2. dataArray
- 3. dataList
- 4. formel
- 5. smallanglescattering (sas)
- 6. formfactor (ff)
- 7. structurefactor (sf)
- 8. dynamic
- 9. dls
- 10. parallel
- 11. Plotting in XmGrace
- 12. mpl
- 13. Examples
- 13.1. In a nutshell
- 13.2. How to build simple models
- 13.3. How to build a more complex model
- 13.4. 1D fits with attributes
- 13.5. 2D fitting
- 13.6. Simple diffusion fit of not so simple diffusion case
- 13.7. How to smooth Xray data and make an inset in the plot
- 13.8. Analyse SAS data
- 13.9. How to fit SANS data including the resolution for different detector distances
- 13.10. Fitting multiple smeared data together
- 13.11. Smearing and desmearing of SAX and SANS data
- 13.12. A long example for diffusion and how to analyze step by step
- 13.13. Sedimentation of two particle sizes and resulting scattering: a Simulation
- 13.14. Create a stacked chart of some curves
- 13.15. A comparison of different dynamic models in frequency domain
- 13.16. Protein incoherent scattering in frequency domain
- 13.17. Fitting a multiShellcylinder in various ways
- 13.18. Hydrodynamic function
- 13.19. Multilamellar Vesicles
- 13.20. 2D oriented scattering
- 13.21. Fitting the 2D scattering of a lattice
- 13.22. A nano cube build of different lattices
- 13.23. Using cloudscattering as fit model
- 13.24. Linear Pearls with connecting coils
- 14. Extending/Contributing/Fortran
- 15. Tips
- 16. Intention and Remarks
- 17. Installation
- 18. Citing Jscatter
- 19. Changelog
Installation¶
Dependencies¶
numpy, scipy -> Mandatory, automatically installed by pip
Cython -> Mandatory, automatically installed by pip
Pillow -> Mandatory, automatic install by pip, for reading of SAXS images
matplotlib -> Mandatory, automatic install by pip, for 3D plots and on Windows
Ipython -> Optional, for convenience as a powerful python shell
gfortran -> Optional, without some functions dont work or use a slower python version
xmgrace -> Optional, preferred plotting on Unix like (use matplotlib on Windows)
Installation of gfortran/xmgrace may need root privileges. Use “sudo” on Linux and MacOS if needed.
Pip installation/upgrade¶
As user (without root access) this will install to your home directory (by default to path ~/.local/). Prepend sudo if it should install to the system python (optional).
pip install jscatter
It is assumed that python3 is your default Python environment and dependencies are installed by a system administrator. You can check python version typing python in the terminal and look at the first lines with version information.
To install dependencies see below.
Check Troubleshooting and tips for help if this is not the case or jscatter is not found after installation.
To upgrade to latest version
pip install jscatter --upgrade
Install from a local repository (development versions):
pip install jscatter --upgrade --pre --find-links /where/the/file/is/saved
options
--user : Install in user directory (folder defined by PYTHONUSERBASE or the default ~/.local)
--find-links : look in the given path for package links e.g development releases
--upgrade : to install upgrades also for dependencies
--pre : to install also development versions
Installation directly from from git repository branch. For different branch use branch name e.g. as development version use dev. (You may need a user account)
pip install git+https://gitlab.com/biehl/jscatter.git@master
Linux¶
Ubuntu, all Debian related
sudo apt-get install build-essential # install gcc and more sudo apt-get install gfortran grace python-matplotlib # or python3-matplotlib e.g. Debian,Fedora, RedHat pip install ipython pip install jscatter
CentOs, Suse, Fedora … do same as above but with yum/zypper…
Manjaro Linux (yaourt asks for permission as root or prepend sudo as above)
# install gfortran yaourt gcc-fortran # install xmgrace (only found in AUR), fonts are needed for the interface, fonts are loaded after restart yaourt xorg-fonts-75 xorg-fonts-100 grace-openmotif python-matplotlib # tk might be missing for tkinter as matplotlib backend sudo pacman -S tk pip install ipython cython pip install jscatter
CONTIN in DLS module (Only if needed).
See DLS module documentation for details how to get and compile the original fortran code.
MacOs (Intel)¶
Homebrew is a MacOs package manager. The following should work on Catalina and BigSur.
Install Homebrew first as given on their web page (see Homebrew). Homebrew installs things to /usr/local/ so you don’t need sudo permissions.
Append the following line to your .zshrc file and restart the terminal:
# set Path to point to homebrew
export PATH=/usr/local/bin:/usr/local/share/python:$PATH
# XQuartz needs this to use flatNamespace correctly with grace (error was something with Vendorshell...)
export DYLD_LIBRARY_PATH=/opt/X11/lib/flat_namespace
Install XQuartz, gcc and xmgrace from Homebrew:
# install XQuartz from homebrew cask (NOT from the AppStore, because of above setting)
brew cask install xquartz
# install xmgrace and gfortran (included in gcc)
brew install gcc grace
# maybe its necessary to relink if old links from previous install are there
brew link --overwrite gcc
brew link --overwrite grace
You may use the system python (omit the next line) or a new version from Homebrew which needs to be installed.
# install python from homebrew (by default python3)
brew install python
Install needed things via pip.
# then use pip (this should call python3 and use pip3)
python3 -m pip install ipython cython
# dependencies are installed automatically
python3 -m pip install jscatter
# check which python is used ( brew: /usr/local/bin/python, MacOS: /usr/local/Library/Frameworks/..../python3 )
which python3
Troubles
If you have trouble like missing xcrun .. you might have to re-register it or update it to the latest version. Try xcode-select –install.
In newer MacOs (e.g. Catalina) you may get an error “stdio.h not found” which is related to the missing /usr/local/include folder as decided by Apple. Add to .zshrc or corresponding to include this path in CPATH:
export CPATH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/
Windows¶
Windows Subsystem for Linux (WSL) with Ubuntu way¶
A new way is based on Windows Subsystem for Linux (WSL), which “lets developers run GNU/Linux environment – including most command-line tools, utilities, and applications – directly on Windows, unmodified, without the overhead of a virtual machine.” (See WSL , use WSL2)
This allows to use any Linux application including XmGrace or Linux editors. You work on the same filesystem as your Windows user account without the need of syncing folders or using a shared folder from a virtual machine. Also gfortran compiled code is working.
The basic procedure is to activate WSL, install Linux and Xserver, add your needed Linux software in the usual way.
Install WSL1 (Please check for the update to WSL2 on following site)
See https://docs.microsoft.com/de-de/windows/wsl/install-win10
Then install a linux distribution of choice from Windows store (tested with Ubuntu 18.04). (The install is per user, so do it under your user account, NOT as administrator)
Open a terminal, start bash and set user and password for Linux. (See https://docs.microsoft.com/en-us/windows/wsl/initialize-distro )
For graphical applications: Install Xserver VcXsrv (or an other Xserver).
See e.g. https://seanthegeek.net/234/graphical-linux-applications-bash-ubuntu-windows/ . Download from https://sourceforge.net/projects/vcxsrv/ . Install (start always manually or add to Windows autostart).
For WSL2 : If you start vcxsrv on windows check in the last option window that “Disable access controll” is checked to allow connections from WSL. If you save the configuration it might remind the settings.
Configure bash to use the local X server
Add this to .bashrc dependent on using WSL1 or WSL2:
export DISPLAY=localhost:0.0 # WSL1 export DISPLAY=$(grep -m 1 nameserver /etc/resolv.conf | awk '{print $2}'):0.0 # WSL2
or run this inside bash with the above line in double quotes “” e.g for WSL1:
echo "export DISPLAY=localhost:0.0" >> ~/.bashrc
Restart bash or load again .bashrc
source ~/.bashrc
Install xmgrace, python3 and Jscatter in Linux subsystem
open terminal, start bash
sudo apt-get update && upgrade # upgrade linux sudo apt-get install python3 python3-pip ipython3 gfortran gcc # python, ipython and pip3 sudo apt-get install numpy python3-matplotlib xmgrace python-tk # mainly for graphical output pip3 install cython pip3 install jscatter # jscatter, scipy, Pillow...
Run test and example.
Anaconda way¶
Anaconda is a python distribution as alternative with numpy, scipy, matplotlib, Ipython preinstalled. Need of sudo depends on how Anaconda was installed (root or user). Maybe the matplotlib backend needs to be configured on Windows to work properly. XmGrace is not working on native Windows.
And there was more to adjust, when i stopped waisting my time. In my testcase it was a pain, but test and examples work (no Xmgrace, no fortran).
I strongly advise to use the WSL way from above, as it is easier to install, update and use. And even gfortran is working in WSL!!
# install jscatter on working anaconda environment
pip install jscatter
Jupyter Notebook¶
Jscatter works on Jupyter Notebooks. You may try Jscatter live at .
The example notebooks are included in the example subfolder of your Jscatter installation or can be downloaded from Gitlab/Jscatter .
To install jscatter on a server Jupyter installation (Jscatter not preinstalled) prepend this on your script
import sys
# install jscatter as user in the current Jupyter kernel
!{sys.executable} -m pip install jscatter
There is some trouble with inline plots (which are not interactive and cannot be updated) dependent on the installed backend. This is related to the used matplotlib backend.
Use this (before importing anything else) to get interactive plots inline.
%matplotlib notebook
Then import Jscatter.
import jscatter as js
js.usempl(True) # use matplotlib
js.usempl(False) # default, use grace on your computer with xmgrace in external window.
If you work over http on a server (usual no display of Xwindows applications) use the same to switch to matplotlib.
Using a cluster/headless mode¶
For long running scripts or fitting one might want to use a cluster.
Cluster allow often no graphical output (as they are used e.g. trough ssh running a script.). To run your script as usual (after testing on your PC) Jscatter implements a headless mode which can be activated using:
import jscatter as js
js.headless(True)
In headless mode all errPlots are saved automatically to lastErrPlot.agr and open matplotlib plots are saved as lastopenedplots_i.png if js.mpl.show() is used. Both allow to retrieve an updated graphical output as stored image instead of updated windows.
Please save important plots in the usual way to store them with a useful name. Additional any data output should be saved in the usual way. Xmgrace is advantageous as .agr format allows later change of the plots including rescaling.
Examples are also adapted and saved to jscatterExamples/example_name/ under the current working directory.
- For experts:
Xmgrace uses gracebat and matplotlib uses the non-interactive backend “Agg”.
For matplotlib create figures using fig=js.mpl.mplot or fig=matplotlib.figure.Figure and later save by fig.savefig. Dont use pyplot as it generates errors in headless mode without interactive loop.
To switch back it is best to restart ipython (at least for matplotlib).
Testing¶
You can test basic functionality of jscatter after installation:
import jscatter as js
js.test.doTest()
#basic graphics and fitting
js.examples.runExample('example_SinusoidalFitting.py')
- The Example shows :
3 sine fit plots with one sine
a fit plot with 5 sine curves fitted simultaneous
a simple plot with 5 points ( phase against Amplitude of the 5 sines)
During development:
python setup.py test
Troubleshooting and tips¶
For testing and playing around without disturbing your productive environment you migth use https://virtualenv.pypa.io .
virtualenv is a tool to create isolated Python environments.
If jscatter is not found after installation then it might be a problem of having multiple versions of python on your system. Starting pip a different version than starting python is the reason.
Using instead of
pip
->pip3
uses the python3 version of pip if old python2 is your default on older systems.Using instead of
pip
->python3 -m pip
in the above commands calls the corrct pip version in any case. This is for example needed if you have a system python which should not be touched and a second python for users.E.g. like this to install also ipython in the same python environment
python3 -m pip install ipython python3 -m pip install jscatter
Latest python2.7 compatible version was 1.1.0 which can be installed by
pip2 install jscatter==1.1.0
If xmgrace is not found by jscatter (
`module 'jscatter' has no attribute 'grace'`
) the path to the executable may be not on your PATH variable.Check this by calling xmgrace in a shell, if it is started it should work.
If xmgrace is not starting, add accordingly the installation path to your PATH variable. For e.g. bash add in your .bashrc (for zsh on MacOs .zshrc) on my CentOS
export PATH=/usr/local/bin:$(PATH)
Restart the shell after changing.
- To open .agr files by klicking add a new file association e.g. to KDE.
In SystemSettings/FileAssociations add a new type xmgrace.
Inside this add FilenamePatterns ‘*.agr’ and similar.
- In ‘ApplicationPreferenceOrder’ add xmgrace and edit this new application :
In General : edit name to “xmgrace” (keep it). In Application : edit name to “xmgracefree” and command to “xmgrace -free”. This will open files in free floating size format.
In ‘ApplicationPreferenceOrder’ add again application xmgrace (no changes) The second opens files in fixed size format (no ‘-free’).