Binary Data, String, and Integer Conversions in Python

In Python 3, struct will interpret bytes as packed binary data:

This module performs conversions between Python values and C structs represented as Python bytes objects. This can be used in handling binary data stored in files or from network connections, among other sources.

>>> import struct
>>> struct.pack("<HH", 2, 8)
b'\x02\x00\x08\x00'
>>> struct.pack("BB", 0x31, 0x32)
b'12'
>>> "12"
'12'

There is no "bytes" type in Python 2. In Python 2, struct will interpret bytes as strings:

This module performs conversions between Python values and C structs represented as Python strings. This can be used in handling binary data stored in files or from network connections, among other sources.

>>> bytes is str

True

>>> struct.pack("<HH", 2, 8)'\x02\x00\x08\x00'
>>> struct.pack("<BB", 0x31, 0x32)'12'>>> "12"'12'

In Python 3,

int.to_bytes(length, byteorder, *, signed=False) will return an array of bytes representing an integer.

>>> a=0x1122334455

>>> b=a.to_bytes(5, byteorder="little")

>>> b.hex()

'5544332211'

int.from_bytes(bytes, byteorder, *, signed=False) will return the integer represented by the given array of bytes.

>>> c = int.from_bytes(b, byteorder="little")
>>> hex(c)
'0x1122334455'

If the integer is within 0~255, you can also use "bytes":

>>> bytes([0x11])

b'\x11'

I couldn't find the built-in method to convert long integer to bytes in Python 2. You can use "binascii" module instead.

String converts to bytes:

>>> s="123456"

>>> s.encode()

b'123456'

Integer converts to String: 

>>> s=hex(a)

>>> s

'0x1122334455'

binascii --- converts between binary and ASCII (string)

binascii.b2a_hex(data)
binascii.hexlify(data)
binascii.a2b_hex(hexstr)
binascii.unhexlify(hexstr)
binascii.crc32(data[, value])

How to View the RSA Verification Data

"""
To view the RSA verification data:
    s = m^d mod n,   and     v = s^e mod n
  =>
    signature = ( message ^ key.D ) mod key.N
    verify (or message) = ( signature ^ key.E ) mod key.N
"""
from Crypto.PublicKey import RSA
from Crypto.Util import number

kf = open('Key.pem', 'r')
myPrivKey = RSA.importKey(kf.read())
kf.close()

fin = open('Signature.bin', 'rb')
signature = bytearray(fin.read())
fin.close()

msg_long = number.bytes_to_long(signature)
veridata = pow(msg_long, myPrivKey.e, myPrivKey.n)
print("%x" %veridata)

Tips of Web Scraping

Tips of Web Scraping

• How to scrape websites with Python and BeautifulSoup
• 手把手 | 范例+代码：一文带你上手Python网页抓取神器BeautifulSoup库
• Python爬蟲新手筆記
• 瀏覽器內的爬蟲初體驗
• Selenium with Python
• 在Windows上安裝Python & Selenium + 簡易教學
• Practical Introduction to Web Scraping in Python
• https://medium.com/@yanweiliu/python%E7%88%AC%E8%9F%B2%E5%AD%B8%E7%BF%92%E7%AD%86%E8%A8%98-%E4%B8%80-beautifulsoup-1ee011df8768

Beautiful Soup:

• Python 使用 Beautiful Soup 抓取與解析網頁資料，開發網路爬蟲教學
• Beautiful Soup網頁解析
• http://blog.castman.net/%E6%95%99%E5%AD%B8/2016/12/22/python-data-science-tutorial-3.html
• https://www.dataquest.io/blog/web-scraping-tutorial-python/

Python Automatic configuration script PAC:

• http://programmersought.com/article/63531430070/
• https://pypac.readthedocs.io/en/latest/user_guide.html

Regular Expressions, RE:

• https://docs.python.org/3/library/re.html
• https://regex101.com/

CSV read/write:

• https://docs.python.org/3/library/csv.html
• https://realpython.com/python-csv/
• https://blog.gtwang.org/programming/python-csv-file-reading-and-writing-tutorial/

認識 Selenium:

"Selenium 是為瀏覽器自動化（Browser Automation）需求所設計的一套工具集合，讓程式可以直接驅動瀏覽器進行各種網站操作。" ...

Python + Selenium:

1. install python
2. install Selenium $pip install selenium
3. install webdriver https://www.seleniumhq.org/about/platforms.jsp

• for Windows
  • download webdriver for Chrome https://sites.google.com/a/chromium.org/chromedriver/
  • 把webdriver解壓縮得到的chromedriver.exe檔放在跟python.exe*同一資料夾內 (ex, *C:\ProgramData\Anaconda3)
• for Linux (or Linux under chromeos):
  • get the appropirate version of chrome driver
  • copy or "chromedriver" (ex, the unzip file of "chromedriver_linux64.zip") to 'usr/local/bin'https://sites.google.com/a/chromium.org/chromedriver/getting-started/chromeos
  • "All ChromeOS test images shall have Chrome Driver binary installed in /usr/local/chromedriver/."* sudo mv chromedriver /usr/local/bin
  • Now, you would need to run something like
    sudo chmod a+x chromedriver to mark it executable.

4. install BeautifulSoup $pip install beautifulsoup

Tips for Python Environment

python help / version (from command line)

> python -h     # List the available parameters 
> python -V 	# show version`

To get a list of locally installed Python modules
>>> help('modules')
or from command line (if you have pip installed):
> pip list or
> python -c help('modules')

How to install a package?
conda example:
> conda install -c cogsci pygame

to install a conda or pip package if it is locally available:
> conda install <package-file-name>.tar.bz2
or
> pip install <package-file-name>.whl

to install a package if the package source code is at the local:
> python setup.py install

conda build

ref:
http://yenlung-blog.logdown.com/posts/257347-anaconda-in-the-virtual-environment-and-package-management

conda version:
> conda -V

update conda itself:
> conda update conda

整個更新Anaconda
> conda update anaconda

Written with StackEdit.

Python on Chromebook

Now you can enable Linux on Chromebook：

https://support.google.com/chromebook/answer/9145439?hl=en

    > setting > Linux（測試版）

Python3 is installed in default.

You can also install anaconda if you prefer:

1. download the corresponding file:
     https://www.anaconda.com/distribution/#download-section

2. run the downloaded shell file:
  ~$ sudo bash ./Anaconda3-2019.03-Linux-x86_64.sh

   Please check the following website for details:
https://chromebook.home.blog/2019/01/20/installing-anaconda-on-a-chromebook-no-dev-beta-or-crouton-needed/

Then, install VSCode from the following site:
https://code.visualstudio.com/
(1. download the deb file, 2. double-click the file to install)

* install GCC:

    $ sudo apt-get install gcc

* how to zip / unzip files:
1. install 7-zip on Linux:
      $ sudo apt-get install p7zip-full
2. zip / unzip commands:
   $ 7z a newfile_name list_of_files
   $ 7z x zipfile_name

Main Function in Python

http://interactivepython.org/courselib/static/thinkcspy/Functions/mainfunction.html

In Python there is nothing special about the name main. We could have called this function anything we wanted. We chose main just to be consistent with some of the other languages.

Before the Python interpreter executes your program, it defines a few special variables. One of those variables is called __name__ and it is automatically set to the string value "__main__" when the program is being executed by itself in a standalone fashion. On the other hand, if the program is being imported by another program, then the __name__ variable is set to the name of that module. This means that we can know whether the program is being run by itself or whether it is being used by another program and based on that observation, we may or may not choose to execute some of the code that we have written.

For example, assume that we have written a collection of functions to do some simple math. We can include a main function to invoke these math functions. It is much more likely, however, that these functions will be imported by another program for some other purpose. In that case, we would not want to execute our main function.

def squareit(n):    
    return n * n

def cubeit(n):    
    return n*n*n

def main():    
    anum = int(input("Please enter a number"))
    print(squareit(anum))
    print(cubeit(anum))

if __name__ == "__main__":
    main()

Random Number

The following code will print 10 pseudo-random integers between 1 and 100

import random
for x in range(10):
  print random.randint(1,101)

For security or cryptographic uses, see the secrets module.
https://docs.python.org/3/library/random.html


random.seed(a=None, version=2)

random.randrange(start, stop[, step])
random.randint(a, b)
random.random()

適合所有中學生及初學者的 Online Judge 系統

https://zerojudge.tw/

因為系統會賦予數量不一的測試資料來測驗您的程式是否正確，因此必須先以一個 while 迴圈來讀取所有的測試資料。

Python example:

import sys

for s in sys.stdin:   

   print('hello, '+s)

If two inputs are required:

import sys
for s in sys.stdin:
    n = s.split()
    a = n[0]
    b = n[1]
    ...