Python Tips
This is my personal memo about useful notations in python. I have not touched on the basics. It is limited to what I find useful.
github
- The jupyter notebook format file on github is here.
google colaboratory
- To run it in google colaboratory here 001/001_nb.ipynb)
Author’s environment
!sw_vers
ProductName: Mac OS X
ProductVersion: 10.14.6
BuildVersion: 18G95
python -V
Python 3.5.5 :: Anaconda, Inc.
timeit
Module for measuring the execution time of functions.
https://docs.python.org/2/library/timeit.html
import timeit
timeit.timeit('[ i for i in range(10)]')
1.7865126989781857
You can specify the number of iterations to take the average. The default is 1000000 (one million times).
number = 1000000
timeit.timeit('[ i for i in range(10)]', number=number)
1.9443903490027878
The timeit.repeat() function can be used to repeat timeiti many times, using the repeat option.
repeat = 5
number = 1000000
timeit.repeat('[ i for i in range(10)]', number=number,repeat=repeat)
[1.810243125015404,
1.9543377529771533,
1.7649507180030923,
1.962000719999196,
1.8034991680178791]
%timeit, %%timeit
This is a magic command to time a process in jupyter notebook format. %%timeit is for the command that is the argument, %%timeit is for the whole cell.
You can also use the -r and -n options to map timeit to repeat and number.
%timeit [i ** 2 for i in range(10000)]]
3.56 ms ± 183 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit -r 5 -n 1000 [i ** 2 for i in range(10000)] ````python
3.51 ms ± 41 µs per loop (mean ± std. dev. of 5 runs, 1000 loops each)
%%timeit
a = [i for i in range(10000)].
b = list(map(lambda x: x **2, a))
4.42 ms ± 260 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
comprehension notation
Creates a list without using for or other methods. The algorithm to create lists is faster and seems to be recommended.
List type.
[i for i in range(5)].
[0, 1, 2, 3, 4].
[i * 2 for i in range(10)] ```python
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
with if
[i * 2 for i in range(10) if i % 2 == 0]
[0, 4, 8, 12, 16] ### if
[i * 2 if i % 2 == 0 else 1 for i in range(10)] ```python
[0, 1, 4, 1, 8, 1, 12, 1, 16, 1]
String is OK.
[ord(i) for i in "TensorFlow"].
[84, 101, 110, 115, 111, 114, 70, 108, 111, 119]
Secondary arrays are also OK.
[[i for i in range(10)] for j in range(10)]
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]
[[j for i in range(10)] for j in range(10)]
[[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2],
[3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3],
[4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4],
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5],
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6],
[7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7],
[8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8],
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9]]
Time measurement
Let’s compare how to create a list using comprehensions notation and the usual, somewhat verbose for statement.
%timeit [i for i in range(1000000)]]
133 ms ± 5.16 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
%%timeit
a = []]
for i in range(1000000):
a.append(i)
202 ms ± 22 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
If we use comprehension notation, we can reduce the time by about 60%. It is better to use comprehension notation for all list types.
Dictionary types
You can also use comprehensions for dictionary types. It is very useful.
a = {'a':1, 'b':2, 'c':3}
print('a : ',a)
print('reverse a : ', {j:i for i,j in a.items()})
a : {'c': 3, 'b': 2, 'a': 1}
reverse a : {1: 'a', 2: 'b', 3: 'c'}
lambda
Basic
This is what is called an anonymous function. It is used for functions that do not need to be given a name. It is rarely used by itself, but is often used with higher-order functions such as map, filter, and sort, as described next.
# No use of the function definition def.
a = lambda x: x ** 2
print(a(10))
print((lambda x: x ** 2)(10))
100
100
With two arguments
### Scalar addition
(lambda a,b: a + b)(1,2)
3
# Add up the list
(lambda a,b: a + b)([1,2,3],[4,5,6])
[1, 2, 3, 4, 5, 6]
if ~ else ~
You can also use if~else~ in lambda.
print((lambda a: a if a == 0 else -100)(-1))
print((lambda a: a if a == 0 else -100)(0))
print((lambda a: a if a == 0 else -100)(1))
-100
0
-100
Higher-order functions
This is a function that includes the function itself as an argument or return value. There are many situations where lambda is used as an argument.
map
The following is an example of usage. For each element in the list, the function is executed uniformly as an argument.
a = [i for i in range(10)].
print('a : ',a)
print('map : '',list(map(lambda x: x ** 2, a)))
a : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].
map : [0, 1, 4, 9, 16, 25, 36, 49, 64, 81].
Here’s an example of an if~else~ in a lambda.
a = [i for i in range(10)].
print('a : ',a)
print('map : '',list(map(lambda x: x ** 2 if x % 3 == 0 else 100, a)))
a : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].
map : [0, 100, 100, 9, 100, 100, 100, 36, 100, 100, 81].
a = [i for i in range(10)].
b = [i for i in range(10,0,-1)]]
print('a : ',a)
print('b : ',b)
print('lambda : ',list(map(lambda a,b: a + b, a,b)))
a : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].
b : [10, 9, 8, 7, 6, 5, 4, 3, 2, 1].
lambda : [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10]
filter
An example usage is as follows. For each element in the list, the function argument will be executed uniformly. Elements with a false result will be removed.
a = [i for i in range(10)].
print('a : ',a)
print('filter1 : '',list(filter(lambda x: x > 5,a)))
print('filter2 : ',list(filter(lambda x: x % 2 == 0,a)))
a : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].
filter1 : [6, 7, 8, 9] filter2 : [0, 2
filter2 : [0, 2, 4, 6, 8].
reduce
The function is defined as reduce(f,x,[op]), where the first argument is a function with two arguments and the second argument is an array. Each element of the array is successively subjected to the first function argument.
import functools
x = [i for i in range(5)].
print('x : ',x)
print('reduce : ',functools.reduce(lambda a,b:a+b, x))
x : [0, 1, 2, 3, 4].
reduce : 10
The order of computation is as follows: 1.
[0,1,2,3,4]=>[0 + 1,2,3,4]=[1,2,3,4]2.[1,2,3,4]=>[1 + 2,3,4]=[3,3,4][3,3,4]=>[3 + 3,4]=[6,4][6,4]=>[6 + 4]=[10]
The final result will be 10.
shutil
There are many ways to use this, and I will update it on a whim.
To delete all files in a directory.
It seems to be a good idea to delete the directory once and create it again from scratch.
import os
import shutil
_dir = '. /test/'
if os.path.exists(_dir):
shutil.rmtree('. /test')
os.mkdir('. /test')
# shutil.rmtree('. /test')
random
This is a module for random numbers.
choice
Randomly selects one element from a given list.
import random
random.choice(range(10))
6
shuffle
Shuffles the list randomly. A destructive method that updates the object itself.
import random
a = [i for i in range(10)].
random.shuffle(a)
print('a : ',a)
a : [0, 9, 3, 7, 4, 1, 5, 2, 8, 6]
sample
Returns a randomly shuffled array of lists. Creates a new object with a non-destructive method.
import random
a = [i for i in range(10)].
b = random.sample(a, len(a))
print('a : ',a)
print('b : ',b)
a : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].
b : [8, 9, 4, 1, 3, 7, 2, 0, 5, 6]
sort
sort method
This is a destructive method. Updates the original object.
import numpy as np
a = list(np.random.randint(10, size=10))
print('before a : ',a)
a.sort()
print('sorted a : ',a)
print('sorted a : ',a) print('sorted a : ',a) ```` before a : [7, 1, 0, 5, 8, 3, 3, 2, 7, 3]
sorted a : [0, 1, 2, 3, 3, 3, 5, 7, 7, 8]
sorted function
This is a non-destructive method. Returns a sorted object.
import numpy as np
a = list(np.random.randint(10, size=10))
print('before a : ',a)
b = sorted(a)
print('sorted a : ',b)
before a : [1, 8, 4, 6, 3, 5, 6, 4, 7, 2].
sorted a : [1, 2, 3, 4, 4, 5, 6, 6, 7, 8].
Sorting lists and objects
The key option is used to specify the elements to sort.
a = [.
['a',1],
['b',6],
['c',3],
['d',2],
]
print('original : ',a)
b = sorted(a,key=lambda x:x[1])
print('sort by ascending order : ', b)
c = sorted(a,key=lambda x:x[1], reverse=True)
print('sort by descending order : ', c)
original : [['a', 1], ['b', 6], ['c', 3], ['d', 2]]
sort by ascending order : [['a', 1], ['d', 2], ['c', 3], ['b', 6]]
sort by descending order : [['b', 6], ['c', 3], ['d', 2], ['a', 1]]
a = [
{'a':1},
{'a':6},
{'a':3},
{'a':2},
]
print('original : ',a)
b = sorted(a,key=lambda x:x['a'])
print('sort by ascending order : ', b)
c = sorted(a,key=lambda x:x['a'], reverse=True)
print('sort by descending order : ', c)
print('sort by descending order : ', c)
original : [{'a': 1}, {'a': 6}, {'a': 3}, {'a': 2}]
sort by ascending order : [{'a': 1}, {'a': 2}, {'a': 3}, {'a': 6}]]
sort by descending order : [{'a': 6}, {'a': 3}, {'a': 2}, {'a': 1}]]
You can also sort and retrieve elements of dictionary types.
a = {
'a':1,
'd':6,
'c':3,
'b':2,
}
print('sort by key')
b = sorted(a.items(), key=lambda x:x[0])
c = sorted(a.items(), key=lambda x:x[0], reverse=True)
print('orig : ',a)
print('asc : ',b)
print('des : ',c)
print()
print('sort by value')
b = sorted(a.items(), key=lambda x:x[1])
c = sorted(a.items(), key=lambda x:x[1], reverse=True)
print('orig : ',a)
print('asc : ',b)
print('des : ',c)
Sort by key
orig : {'b': 2, 'c': 3, 'd': 6, 'a': 1}
asc : [('a', 1), ('b', 2), ('c', 3), ('d', 6)]
des : [('d', 6), ('c', 3), ('b', 2), ('a', 1)]
Sort by value
orig : {'b': 2, 'c': 3, 'd': 6, 'a': 1}
asc : [('a', 1), ('b', 2), ('c', 3), ('d', 6)]
des : [('d', 6), ('c', 3), ('b', 2), ('a', 1)]
Other
Cast a set type list to a dict type
Note that this is useful for some things.
a = [
("a", 1),
("b", 2),
("c", 3),
("d", 4),
]
dict(a)
{'a': 1, 'b': 2, 'c': 3, 'd': 4}