Python Misc

Network Programming: Network byte order is big endian…This is the natural number order. Intel system is little endian.(little end first) print socket.gethostbyaddr('8.8.8.8') print socket.gethostbyname('google.com') pcapy is python library for capturing network packets. It can be used to analyse offline pcap files as well. You can create a raw packet with ‘scapy’

High Performance Python

CPU can do multiple operation at the same time with no additional cost on time or performance. Use vectorization to utilize this. possible with libraries such as 'numpy'.

Design your code in such a way that to minimize CPU moving data from RAM to L2 cache and minimize number of reads CPU has to perform from RAM.

Natively python can't make use of multiple CPU core. Inorder to avoid this problem use multiprocessing instead of multiple threads.

line_profiler profile a given function line by line.
heapy a track all objects in Python memory.
perf stat  to see number of instructions ultimately executed by CPU.
memory_profile to show memory usage.

#python -m timeit -n 5 -r 5 "import test"
5 loops, best of 5: 0.763 usec per loop

simple timing using unix time command : time -p python test.py

or you can use cProfile

$ python -m cProfile -s cumulative test.py

         3 function calls in 0.002 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.002    0.002    0.002    0.002 test.py:1(<module>)
        1    0.000    0.000    0.000    0.000 {range}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

This gives cumulative time of each function

within the code, you can use 'import timeit' module for timing.

Intro to Webservers for Python

When a user enters a web site, their browser makes a connection to the site’s web server (this is called the request). The server looks up the file in the file system and sends it back to the user’s browser, which displays it (this is the response). This is roughly how the underlying protocol, HTTP, works.

Dynamic web sites are not based on files in the file system, but rather on programs which are run by the web server when a request comes in, and which generate the content that is returned to the user.

They can do all sorts of useful things, like display the postings of a bulletin board, show your email, configure software, or just display the current time. These programs can be written in any programming language the server supports. Since most servers support Python, it is easy to use Python to create dynamic web sites.

Most HTTP servers are written in C or C++, so they cannot execute Python code directly – a bridge is needed between the server and the program. These bridges, or rather interfaces, define how programs interact with the server. There have been numerous attempts to create the best possible interface, but there are only a few worth mentioning.

Not every web server supports every interface. Many web servers only support old, now-obsolete interfaces; however, they can often be extended using third-party modules to support newer ones.

Common Gateway Interface (CGI):

This interface, most commonly referred to as “CGI”, is the oldest, and is supported by nearly every web server out of the box. Programs using CGI to communicate with their web server need to be started by the server for every request. So, every request starts a new Python interpreter – which takes some time to start up – thus making the whole interface only usable for low load situations.

The upside of CGI is that it is simple – writing a Python program which uses CGI is a matter of about three lines of code. This simplicity comes at a price: it does very few things to help the developer.

Writing CGI programs, while still possible, is no longer recommended.

Web Server Gateway Interface (WSGI):

The Web Server Gateway Interface, or WSGI for short, is defined in PEP 333 and is currently the best way to do Python web programming.

WSGI is the Web Server Gateway Interface. It is a specification that describes how a web server communicates with web applications, and how web applications can be chained together to process one request.

While it is great for programmers writing frameworks, a normal web developer does not need to get in direct contact with it. When choosing a framework for web development it is a good idea to choose one which supports WSGI.

The big benefit of WSGI is the unification of the application programming interface. When your program is compatible with WSGI – which at the outer level means that the framework you are using has support for WSGI – your program can be deployed via any web server interface for which there are WSGI wrappers.

A really great WSGI feature is middleware. Middleware is a layer around your program which can add various functionality to it. There is quite a bit of middleware already available.

For example, instead of writing your own session management (HTTP is a stateless protocol, so to associate multiple HTTP requests with a single user your application must create and manage such state via a session), you can just download middleware which does that, plug it in, and get on with coding the unique parts of your application.

The same thing with compression – there is existing middleware which handles compressing your HTML using gzip to save on your server’s bandwidth.

Authentication is another a problem easily solved using existing middleware.

Data Persistance:

Relational databases are queried using a language called SQL. Python programmers in general do not like SQL too much, as they prefer to work with objects. It is possible to save Python objects into a database using a technology called ORM (Object Relational Mapping).

ORM translates all object-oriented access into SQL code under the hood, so the developer does not need to think about it. Most frameworks use ORMs, and it works quite well.

Credit:https://docs.python.org/2/howto/webservers.html