Archive for August, 2019

Running httpbin on Azure

August 24, 2019

Wait but why? Rationale

The purpose of this exercise is to deploy an adhoc server for testing purposes. Advantages of this are:

  • Configurability of testing service (we can extend it however we like)
  • Reliability / control over testing service

Create Azure VM

First, create an Azure VM with Ubuntu 18.04, making sure you provide ssh access.

In particular:

  • Select B1s. This gives you 1GB of ram and 1CPU, which should be sufficient
  • Make sure you provide yourself with ssh access by providing your ~/.ssh/id_rsa.pub to the machine
  • Expose ports 22, 443 and 80
  • Make sure you disable automatic shutdown. If you don’t your IP will cycle, which will foul DNS resolution.

Select and configure a Freenom domain

Get a freenom DNS and point it to your IP. (I created httpbin.ga.)

Install docker and build a custom image

Then ssh into your box

sudo apt-get update
sudo apt install -y docker.io

Note: if we merely wanted to serve on http rather than https, we could be done at this point by simply running “sudo docker pull kennethreitz/httpbin & sudo docker -d -p 80:80 kennethreitz/httpbin”, and not need to build a custom image. But we want to deploy a testing server, so we need https as well. As is always the case with these things, it turns out 80-90% of the effort is in adding a small additional (but important) piece of functionality.

Clone kennethreitz/httpbin, alter the Dockerfile to expose 54 and run on 54. We need to do this because certbot requires port 80.

You will also need a docker hub id, and to log into your docker hub account before doing this step.

docker build -t your-docker-hub-id/httpbin .
docker push your-docker-hub-id/httpbin

Then run your custom image on your server,

docker pull your-docker-hub-id/httpbin
docker run -d -p 80:54 your-docker-hub-id/httpbin

Check that things work by going to your VM ip and seeing that the application loads. Then

docker ps
docker stop <container_id>
docker rm <container_id>
docker run -d -p 42:54 your-docker-hub-id/httpbin 
# port 42 of host goes to port 54 of container

Install Apache

Install apache and start it up.

sudo apt install -y apache2

Set up a domain on apache (use the freenom domain you set up before). Mine is httpbin.ga. You should replace this with your own domain.

sudo mkdir /var/www/httpbin.ga
sudo chown -R $USER:$USER /var/www/httpbin.ga
sudo chmod -R 755 /var/www/httpbin.ga
nano /var/www/httpbin.ga/index.html
<html>
    <head>
        <title>Welcome to httpbin.ga!</title>
    </head>
    <body>
        <h1>Success!  The httpbin.ga virtual host is working!</h1>
    </body>
</html>
sudo nano /etc/apache2/sites-available/httpbin.ga.conf
<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    ServerName httpbin.ga
    ServerAlias www.httpbin.ga
    DocumentRoot /var/www/httpbin.ga
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
sudo a2ensite httpbin.ga.conf
sudo a2dissite 000-default.conf
sudo apache2ctl configtest
sudo systemctl restart apache2

Set up certbot

Install certbot to secure your domain name. Certbot allows us to automatically refresh our site certificate with LetsEncrypt, so that we are always secured.

sudo add-apt-repository ppa:certbot/certbot
sudo apt install -y python-certbot-apache
sudo certbot --apache -d httpbin.ga -d www.httpbin.ga

Select 1 ‘no-redirect’.

Do a dry-run of certbot’s auto-renew functionality.

sudo certbot renew --dry-run

Serve the application on http

Now, we’d like to serve our docker application.

First, make sure that port 42 is open on your Azure VM. This is very important, since otherwise the next step won’t work.

In your apache2.conf, set up forwarding to the correct port

sudo vim /etc/apache2/sites-available/httpbin.ga.conf
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
<VirtualHost *:80>
    ProxyPreserveHost On
    ProxyRequests Off
    ServerAdmin webmaster@localhost
    ServerName httpbin.ga
    ServerAlias www.httpbin.ga
    DocumentRoot /var/www/httpbin.ga
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
    ProxyPass /app http://localhost:42/
    ProxyPassReverse /app http://localhost:42/
</VirtualHost>

then

a2enmod proxy_http

Now, navigate to your.domain/app, and you should see the application, albeit with no css or a few other things.

Attempt to serve the application on https

Ok, almost there! Now we want to

  • modify the application docker image again, to fix the css errors (since we are now serving on /app, rather than the base domain)
  • allow ourselves to redirect on https, so that https://your.domain/app works, too.
  • Create two new logging files, ${APACHE_LOG_DIR}/443_error.log and ${APACHE_LOG_DIR}/443_access.log
sudo touch /var/log/apache2/443_error.log
sudo touch /var/log/apache2/443_access.log

Enable things we will need for ssl

sudo a2enmod rewrite
sudo a2enmod ssl

Update our apache.conf file

sudo vim /etc/apache2/sites-available/httpbin.ga.conf
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so

<VirtualHost *:80>
    # ProxyPreserveHost On

    # server config
    ServerAdmin webmaster@localhost
    ServerAlias www.httpbin.ga
    ServerName httpbin.ga
    DocumentRoot /var/www/httpbin.ga

    # proxy
    ProxyRequests Off
    ProxyPass /app http://localhost:42/
    ProxyPassReverse /app http://localhost:42/

    # logs
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

<VirtualHost *:443>
    # ssl
    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/httpbin.ga/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/httpbin.ga/privkey.pem

    # server config
    ServerAdmin webmaster@localhost
    ServerAlias www.httpbin.ga
    ServerName httpbin.ga
    DocumentRoot /var/www/httpbin.ga

    # proxy
    ProxyRequests Off
    ProxyPass /app http://localhost:42/
    ProxyPassReverse /app http://localhost:42/

    # logs
    LogLevel debug
    ErrorLog ${APACHE_LOG_DIR}/443_error.log
    CustomLog ${APACHE_LOG_DIR}/443_access.log combined
</VirtualHost>
sudo chown -R $APACHE_RUN_USER:$APACHE_RUN_GROUP /etc/letsencrypt/live/httpbin.ga
sudo apache2ctl configtest
sudo systemctl restart apache2

This is enough to serve on your.site/app with http, but not securely at https://your.site/app with https – you will get 404 not found. The solution is to use nginx as a reverse proxy for apache .

Set up Nginx as a Reverse proxy for apache

First, install nginx

sudo apt install -y nginx

* install nginx https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04 (sudo systemctl start ufw)
* then secure with let-s encrypt and renew automatically via cron job https://www.digitalocean.com/community/tutorials/how-to-secure-apache-with-let-s-encrypt-on-ubuntu-16-04

Finally, DNS! Maybe will update later. Still haven’t been able to properly progress beyond http only (no certificate) and running httpbin on the basic ip. Would need to use FreeNom to get a domain to use and have A records and point to the correct nameservers on said service.

Water in arid places: Reticular Chemistry, MOFs, COFs, weaving and pathways to practical nanotechnology

August 10, 2019

Overview

I recently noticed that New Scientist made mention of a particular application of some particularly interesting research directed by Omar Yaghi, an American-Jordanian chemist at UC Berkeley. This work is notable for a number of reasons. Perhaps the best introduction to it is this video, here, which was recorded at Stockholm University May 9-10, 2019 earlier this year:

Notes from the talk

In the video, a few notable things are mentioned:

  • Up until the 1990s – indeed, 1993:
    • polymer chemistry (1d chains) was state of the art
    • 0d (organic chemistry) was well understood
    • 2d, 3d chemistry was not well understood
  • The basic game of chemistry is to create compounds that aid synthesis of other compounds
    • In order to have reproducible results you need to crystallise your catalytic compounds
    • But the crystallisation problem was not solved
    • A key problem with crystallisation is understanding how to deal with compounds with bonds that are stronger than Van de Waals forces, Hydrogen bonds, and M-donor bonds in coordination networks

In the late 1990s, Omar made progress with reticular chemistry with MOF (metal organic frameworks) that solved the crystallisation problem. These used M-charged linker bonds.

Later still, in the 2000-2010 decade, Omar made progress with Covalent organic framework using covalent bonds.

Together, these two forms of stronger bonds form the basis of what he calls Reticular Chemistry.

Generalisations of COFs include molecular weaving (~2016). Current research involves using a mixture of heterogeneity in crystalline backbone (rather like DNA) to mimic proteins (i.e. organic molecular machines) in a much wider variety of reaction conditions. i.e., nanotechnology.

But the rest of his talk didn’t really focus on his work from 2000+; rather, it focused on applications of the pure research on MOFs dating from the 90s and its gearing for practical applications (a company based on these applications, Water Harvesting Inc. (http://www.wahainc.com/) is due to launch in October this year).

Basically his point was that with MOFs you can build a massive massive amount of combinatorial possibilities of catalytic compounds that can do various things, depending on:

  • the choice of metal
  • the choice of organic molecule
  • the choice of geometry

These form a ‘periodic table’ of reticular chemistry of tremendous combinatorial complexity. “[If you can imagine it, you can build it]” paraphrasing Omar in his talk.

Possibilities of applications of such combinatorial choices include:

  • hydrogen storage (for hydrogen powered fuel cells)
  • methane storage (up to 3 times that of just storing the methane without the MOF)
  • carbon dioxide sequestration and conversion into methanol fuel (work in progress, currently in the lab, but with obvious applications to combating global warming while producing a useful product to boot)
  • water generation in arid air

He spoke a bit about water generation. There are apparently 6 septillion litres of water in the air at any one time – as much as is present in freshwater lakes and streams. Apparently a lot of current desalination technologies rely on high humidity (~65%) and also for the air to be cooled to ~1.5 degrees celsius for the Carnot cycle / Dew point to be reached. So 1) very energy inefficient and 2) not typical of desert conditions, where humidity might be between 5% and 25-30% at most during the night.

His team ultimately created a device that served as the desiccant, i.e. with the MOF as the desiccant. It required no power – just sunlight. And it generated (first prototype with zirconium) about 0.2L of water per day per kg of MOF. The second prototype (since zirconium is about $150 USD / kg) used aluminium, and produced about 1L of water per kg per day.

Bottom line

Water Harvesting Inc. is launching passive water extractors from low humidity air in October; they (wahainc) are looking to commercialise this technology, and release very soon. A tremendous success story and significant for all sorts of reasons; if nothing else because:

  • This rudimentary application of basic nanotechnology has the potential to solve water problems in water stressed regions
  • This technique promises fresh water where potable water is otherwise not present
  • This technique also provides a potential route to carbon dioxide sequestration and mitigation of global warming

More excitingly, the future theoretical direction of this research seems like a nice natural pathway to look into designing customisable and specialised molecular machines for performing very specific tasks. Not programmable / adaptive machines by any means – that would be assembler level control – but one could certainly potentially imagine the ability to program heterogeneity into a crystalline MOF or COF to perform a specialised operation at one further level of abstraction, so that one would be in essence building reaction vessels at runtime in order to build something more complicated at molecular level. That would be an assembler, or a pathway to creating assembler prototypes.

References