[ ]
ZhouSa.com-周飒博客 1. 在Ipython Notebook中的代码调试与优化
jupyter 是科学计算工具,那代码的调优就是它的一个重点了,python本身的运算能力其实很令人着急的,但通过分析计算瓶颈和用numpy,cython等工具优化代码,python也可以拥有非常高的运算效率(其实是C的功劳)
本文的先验知识有:
1.1. 调试代码
python报异常从来都是一大段,很难看也很难看懂,可以使用%xmode Plain和%xmode Verbose来在精简模式和运来模式间切换
def f1(a,b): return a/b def f2(x): a = x b = x-1 return f1(a,b) %xmode Plain Exception reporting mode: Plain f2(1) Traceback (most recent call last): File "<ipython-input-3-d9076a5554c7>", line 1, in <module> f2(1) File "<ipython-input-1-d7ac5604b6da>", line 6, in f2 return f1(a,b) File "<ipython-input-1-d7ac5604b6da>", line 2, in f1 return a/b ZeroDivisionError: division by zero %xmode Verbose Exception reporting mode: Verbose f2(1) --------------------------------------------------------------------------- ZeroDivisionError Traceback (most recent call last) <ipython-input-5-d9076a5554c7> in <module>() ----> 1 f2(1) global f2 = <function f2 at 0x106a34bf8> <ipython-input-1-d7ac5604b6da> in f2(x=1) 4 a = x 5 b = x-1 ----> 6 return f1(a,b) global f1 = <function f1 at 0x106a34840> a = 1 b = 0 <ipython-input-1-d7ac5604b6da> in f1(a=1, b=0) 1 def f1(a,b): ----> 2 return a/b a = 1 b = 0 3 def f2(x): 4 a = x 5 b = x-1 ZeroDivisionError: division by zero 使用%debug会在报错时进去调试模式,在调试模式中我们可以
- 输入变量名来获取变量的情况,
- 输入up来进入上一层查看
要退出输入quit即可
1.1.3. 运行时间检查
计算机再怎么算的慢也是比人快的多的,人的直觉并不能很好的感知到一个程序运行的快不快慢不慢,这种时候就要用时间检查工具. ipython中常用的就是
%timeit <func>命令了
%timeit sum(map(lambda x:x**2,range(10000000))) %xmode Plain 0或者查看具体时间在哪里损耗的%time
%xmode Plain 1%xmode Plain 2%prun将会产生一个有序表格来展示在该语句中所调用的每个内部函数调用的次数，每次调用的时间与该函数累计运行的时间。
%xmode Plain 3在ipython中使用line_profiler可以使用他们的ipython魔法命令%lprun,要使用这个魔法命令需要先加载
%lprun可以带上参数 -f 指定需要检查的方法
%xmode Plain 4%xmode Plain 5与line_profiler类似,memory_profiler也有对ipython的支持,使用方式也类似,他的魔法命令是%menit和%mprun
关于内存的使用,在数据量小的时候看不出来但一旦数据量大了就会很棘手,ipython中查看内存使用的魔法命令是%menit
%xmode Plain 6%xmode Plain 7%xmode Plain 8%mprun只能检查引用进来的模块的内存性能,因此需要先将代码写到文件重作为模块引入
%mprun可以带上参数 -f 指定需要检查的方法
%xmode Plain 9Exception reporting mode: Plain 01.2. *使用C语言扩展做代码优化
对于提高python运行速度,我们常用C语言来加速,对于用C语言构建核心运算模块,Cython是numpy,scipy的发展方向.一般用Cython我们都是拿他写模块,写好后要编译安装,而ipython notebook对Cython有相当好的支持
我们可以用%load_ext Cython来直接编译运行Cython写出来的程序
Exception reporting mode: Plain 1我们以斐波那契数列来举例,看看用在ipython中python的速度可以有多快
1.2.1. 用Cython优化性能
- 原版python
Exception reporting mode: Plain 2Exception reporting mode: Plain 3Exception reporting mode: Plain 4Exception reporting mode: Plain 5Exception reporting mode: Plain 6- 直接用cython加速
Exception reporting mode: Plain 7Exception reporting mode: Plain 8Exception reporting mode: Plain 4f2(1) 0f2(1) 1时间上和原版俩差了3倍的速度
- 使用静态编译
f2(1) 2f2(1) 3Exception reporting mode: Plain 4f2(1) 5f2(1) 6速度直线上升,快了100倍不止!
- 使用缓存计算(递归改迭代)
f2(1) 7f2(1) 8f2(1) 9Traceback (most recent call last): File "<ipython-input-3-d9076a5554c7>", line 1, in <module> f2(1) File "<ipython-input-1-d7ac5604b6da>", line 6, in f2 return f1(a,b) File "<ipython-input-1-d7ac5604b6da>", line 2, in f1 return a/b ZeroDivisionError: division by zero 0或者简单的使用变量
Traceback (most recent call last): File "<ipython-input-3-d9076a5554c7>", line 1, in <module> f2(1) File "<ipython-input-1-d7ac5604b6da>", line 6, in f2 return f1(a,b) File "<ipython-input-1-d7ac5604b6da>", line 2, in f1 return a/b ZeroDivisionError: division by zero 1Traceback (most recent call last): File "<ipython-input-3-d9076a5554c7>", line 1, in <module> f2(1) File "<ipython-input-1-d7ac5604b6da>", line 6, in f2 return f1(a,b) File "<ipython-input-1-d7ac5604b6da>", line 2, in f1 return a/b ZeroDivisionError: division by zero 2Exception reporting mode: Plain 4Traceback (most recent call last): File "<ipython-input-3-d9076a5554c7>", line 1, in <module> f2(1) File "<ipython-input-1-d7ac5604b6da>", line 6, in f2 return f1(a,b) File "<ipython-input-1-d7ac5604b6da>", line 2, in f1 return a/b ZeroDivisionError: division by zero 4Traceback (most recent call last): File "<ipython-input-3-d9076a5554c7>", line 1, in <module> f2(1) File "<ipython-input-1-d7ac5604b6da>", line 6, in f2 return f1(a,b) File "<ipython-input-1-d7ac5604b6da>", line 2, in f1 return a/b ZeroDivisionError: division by zero 5原版的Python对迭代的优化还是相当可以的利用两个变量存储过程量,可以大大减少运算量
- 使用缓存并且使用Cython加速
Traceback (most recent call last): File "<ipython-input-3-d9076a5554c7>", line 1, in <module> f2(1) File "<ipython-input-1-d7ac5604b6da>", line 6, in f2 return f1(a,b) File "<ipython-input-1-d7ac5604b6da>", line 2, in f1 return a/b ZeroDivisionError: division by zero 6Traceback (most recent call last): File "<ipython-input-3-d9076a5554c7>", line 1, in <module> f2(1) File "<ipython-input-1-d7ac5604b6da>", line 6, in f2 return f1(a,b) File "<ipython-input-1-d7ac5604b6da>", line 2, in f1 return a/b ZeroDivisionError: division by zero 7Traceback (most recent call last): File "<ipython-input-3-d9076a5554c7>", line 1, in <module> f2(1) File "<ipython-input-1-d7ac5604b6da>", line 6, in f2 return f1(a,b) File "<ipython-input-1-d7ac5604b6da>", line 2, in f1 return a/b ZeroDivisionError: division by zero 8Traceback (most recent call last): File "<ipython-input-3-d9076a5554c7>", line 1, in <module> f2(1) File "<ipython-input-1-d7ac5604b6da>", line 6, in f2 return f1(a,b) File "<ipython-input-1-d7ac5604b6da>", line 2, in f1 return a/b ZeroDivisionError: division by zero 9%xmode Verbose 0Exception reporting mode: Plain 4%xmode Verbose 2%xmode Verbose 3惊了...只有1微秒左右!
- 再静态化
%xmode Verbose 4%xmode Verbose 5Exception reporting mode: Plain 4%xmode Verbose 7%xmode Verbose 8又快了4倍
1.2.2. 使用numba加速
%xmode Verbose 9Exception reporting mode: Verbose 0Exception reporting mode: Verbose 1Exception reporting mode: Plain 4Exception reporting mode: Verbose 3Exception reporting mode: Verbose 4略不如Cython的最终版本
还没有评论，来说两句吧...