Fourinone和hadoop运行wordcount的对比测试（平均4核4g配置，输入数据为文件）：

说明：Fourinone可以充分利用单机并行能力，4核计算机可以4个并行实例计算，hadoop目前只能N*1；另外，可以由上图看出，如果要完成20g的数据，实际上fourinone只需要使用5台机器用60秒完成，比使用19台机器完成19g的hadoop节省了14台机器，并提前了200多秒

相关分布式框架(jar格式)及demo代码下载：

http://download.csdn.net/detail/fourinone/3557912

http://www.skycn.com/soft/68321.html

作者介绍：
Stone.Peng
资深IT技术人士
现在淘宝网任高级专家，从事互联网核心技术研究
之前在金蝶总体架构部任SOA架构师，负责设计ESB

您可能也喜欢：

五个最佳的Hadoop项目

淘宝在数据处理领域的项目及开源产品介绍

代码调优法则--编程珠玑笔记

TF-IDF及文本相似性度量

数据完整性策略

无觅

在一份给定的文件里，词频（term frequency，TF）指的是某一个给定的词语在该文件中出现的次数。这个数字通常会被正规化，以防止它偏向长的文件。（同一个词语在长文件里可能会 比短文件有更高的词频，而不管该词语重要与否。）对于在某一特定文件里的词语 t_i 来说，它的重要性可表示为：

以上式子中 n_i,j 是该词在文件d_j中的出现次数，而分 母则是在文件d_j中所有字词的出现次数 之和。

逆向文件频率（inverse document frequency，IDF）是一个词语普遍重要性的度量。某一特定词语的IDF，可以由总文件数目除以包含该词语之文件的数目，再将得到的商取对数得到：

其中

• |D|：语料库中的文件总数
• ： 包含词语t_i的文件数目（即的 文件数目）

然后

某一特定文件内的高词语频率，以及该词语在整个文件集合中的低文件频率，可以产生出高权重的TF-IDF。因此，TF-IDF倾向于过滤掉常见的词 语，保留重要的词语。

=================文本相似性度量=======================

方法一：向量空间模型

在向量空间模型中，文本泛指各种机器可读的记录。用D（Document）表示，特征项（Term，用t表示）是指出现在文档D中且能够代表该文档内容的 基本语言单位，主要是由词或者短语构成，文本可以用特征项集表示为D(T1，T2，…，Tn)，其中Tk是特征项，1<=k<=N。例如一篇 文档中有a、b、c、d四个特征项，那么这篇文档就可以表示为D(a，b，c，d)。对含有n个特征项的文本而言，通常会给每个特征项赋予一定的权重表示 其重要程度。即D＝D(T1，W1；T2，W2；…，Tn，Wn)，简记为D＝D(W1，W2，…，Wn)，我们把它叫做文本D的向量表示。其中Wk是 Tk的权重，1<=k<=N。在上面那个例子中，假设a、b、c、d的权重分别为30，20，20，10，那么该文本的向量表示为 D(30，20，20，10)。在向量空间模型中，两个文本D1和D2之间的内容相关度Sim(D1，D2)常用向量之间夹角的余弦值表示，公式为：

其 中，W1k、W2k分别表示文本D1和D2第K个特征项的权值，1<=k<=N。
在自动归类中，我们可以利用类似的方法来计算待归类 文档和某类目的相关度。例如文本D1的特征项为a，b，c，d，权值分别为30，20，20，10，类目C1的特征项为a，c，d，e，权值分别为 40，30，20，10，则D1的向量表示为D1(30,20,20,10,0),C1的向量表示为C1（40，0，30，20，10），则根据上式计算 出来的文本D1与类目C1相关度是0.86

方法二：字符串相似度

对于象字符串计算相似度的算法有很多，常用的有最大公共字串，编辑距离等。

编辑距离就是用来计算从原串（s）转换到目标串(t)所需要的最少的插入，删除和替换的数目，在NLP中应用比较广泛，如一些评测方法中就用到了 （wer,mWer等），同时也常用来计算你对原文本所作的改动数。编辑距离的算法是首先由俄国科学家Levenshtein提出的，故又叫 Levenshtein Distance。

您可能也喜欢：

淘宝在数据处理领域的项目及开源产品介绍

代码调优法则--编程珠玑笔记

开发环境的三大规则

软件构建中的理想设计特征

会话状态模式

无觅

2012看来没有世界末日的迹象，日子还是得过。2012事情很多，要静下心来做一件自己想做而没有做的事情，可能很难，希望能坚持。2012我希望能够将持续集成引入自己的项目中，让自动化代替部分人工，测试的同事太辛苦，也为了让自己睡个安稳觉。2012年依然有值得期待的书籍，读10本书是必须的。2012得换个手机，旧手机一直没坏，换掉觉得浪费。2012要多认识些朋友，都世界末日了，在黄泉路上大家有个伴。2012多活跃一些，多写点博客。2012希望下班后能早点回家。2012希望自己和亲人身体都好，这才是革命的本钱。

您可能也喜欢：

又过去了一年

ChinaUnix网络优化论坛上海站总结

个人学习方法总结

迟到的2009总结

80 years

无觅

简介
基本原理与目标
概述
应用接口
服务器接口
中间件 : 同时扮演两种角色的组件
详细说明
environ 变量
输入和错误流
start_response() 可调用者
Handling the Content-Length Header
Buffering and Streaming
Middleware Handling of Block Boundaries
The write() Callable
Unicode Issues
Error Handling
HTTP 1.1 Expect/Continue
Other HTTP Features
Thread Support
Implementation/Application Notes
Server Extension APIs
Application Configuration
URL Reconstruction
Supporting Older (<2.2) Versions of Python
Optional Platform-Specific File Handling
Questions and Answers
Proposed/Under Discussion
Acknowledgements
References
Copyright
简介

本文档描述一份在web服务器与web应用/web框架之间的标准接口，此接口的目的是使得web应用在不同web服务器之间具有可移植性。

基本原理与目标

python目前拥有大量的web框架，比如 Zope, Quixote, Webware, SkunkWeb, PSO, 和Twisted Web。大量的选择使得新手无所适从，因为总得来说，框架的选择都会限制web服务器的选择。

对比之下，虽然java也拥有许多web框架，但是java的" servlet" API使得使用任何框架编写出来的应用程序可以在任何支持" servlet" API的web服务器上运行。 服务器中这种针对python的API（不管服务器是用python写的，还是内嵌python，还是通过一种协议来启动python）的使用和普及，将分离人们对web框架和对web服务器的选择，用户可以自由选择适合他们的组合，而web服务器和web框架的开发者也能够把精力集中到各自的领域。

因此，这份PEP建议在web服务器和web应用/web框架之间建立一种简单的通用的接口规范，Python Web Server Gateway Interface (WSGI).

但是光有这么一份规范对于改变web服务器和web应用/框架的现状是不够的，只有web服务器和web框架的作者们实现WSGI，他才能起应有的效果。

然而，既然还没有任何框架或服务器实现了WSGI，对实现WSGI也没有什么直接的奖励，那么WSGI必须容易实现，这样才能降低作者的初始投资。

服务器和框架两边接口的实现的简单性，对于WSGI的作用来说，绝对是非常重要的。所以这一点是任何设计决策的首要依据。

对于框架作者来说，实现的简单和使用的方便是不一样的。WSGI为框架作者展示一套绝对没有"frills"的接口，因为象response对象和对cookie的处理这些问题和框架现有的对这些问题的处理是矛盾的。再次重申一遍，WSGI的目的是使得web框架和web服务器之间轻松互连，而不是创建一套新的web框架。

同时也要注意到，这个目标使得WSGI不能依赖任何在当前已部署版本的python没有提供的任何功能，因此，也不会依赖于任何新的标准模块，并且WSGI并不需要2.2.2以上版本的python(当然，在以后的python标准库中内建支持这份接口的web服务器也是个不错的主意)

不光要让现有的或将要出现的框架和服务器容易实现，也应该容易创建请求预处理器、响应处理程序和其他基于WSGI的中间件组件，对于服务器来说他们是应用程序，而对于他们包含的应用程序来说他们是服务器。

如果中间件既简单又健壮，而且WSGI广泛得实现在服务器和框架中，那么就有可能出现全新的python web框架：整个框架都是由几个WSGI中间件组件组成。甚至现有框架的作者都会选择重构将以实现的服务以这种方式提供，变得更象一些和WSGI配合使用的库而不是一个独立的框架。这样web应用开发这就可以根据特定功能选择最适合的组件，而不是所有功能都由同一个框架提供。

当然，这一天无疑还要等很久，在这之间，一个合适的短期目标就是让任何框架在任何服务器上运行起来。

最后，需要指出的是当前版本的WSGI并没有规定一个应用具体以何种方式部署在web服务器或gateway上。目前，这个需要由服务器或gateway的具体实现来定义。如果足够多实现了WSGI的服务器或gateway通过领域实践产生了这个需求，也许可以产生另一份PEP来描述WSGI服务器和应用框架的部署标准。

概述

WSGI接口有两种形式：一个是针对服务器或gateway的，另一个针对应用程序或框架。服务器接口请求一个由应用接口提供的可调用的对象，至于该对象是如何被请求的取决与服务器或gateway。我们假定一些服务器或gateway会需要应用程序的部署人员编写一个简短的脚本来启动一个服务器或gateway的实例，并把应用程序对象提供得服务器，而其他的服务器或gateway需要配置文件或其他机制来指定从哪里导入或者或得应用程序对象。

除了纯粹的服务器/gateway和应用程序/框架，还可以创建实现了这份规格说明书的中间件组件，对于包含他们的服务器他们是应用程序，而对于他们包含的应用程序来说他们是服务器，他们可以用来提供可扩展的API，内容转换，导航和其他有用的功能。

在整个规格说明书中，我们使用短语"一个可调用者"意思是"一个函数，方法，类，或者拥有 __call__ 方法的一个对象实例",这取决与服务器，gateway，应用程序根据需要而选择的合适实现方式。相反服务器，gateway和请求一个可调用者的应用程序不可以依赖具体的实现方式，not introspected upon.

应用接口

一个应用程序对象是一个简单的接受两个参数的可调用对象，这里的对象并不是真的需要一个对象实例，一个函数、方法、类、或者带有 __call__ 方法的对象实例都可以用来做应用程序对象。应用程序对象必须可以多次被请求，实际上服务器/gateway(而非CGI)确实会产生这样的重复请求。

(注意：虽然我们把他叫做"应用程序"对象，但并不是说程序员要把WSGI当做API来调用，我们假定应用程序开发者仍然使用更高层面上的框架服务来开发应用程序，WSGI是提供给框架和服务器开发者使用的工具，并不打算直接对应用程序开发者提供支持)

这里有两个应用程序对象的示例，一个是函数，另一个是类:

def simple_app(environ, start_response):
"""也许是最简单的应用程序对象"""
status = '200 OK'
response_headers = [('Content-type','text/plain')]
start_response(status, response_headers)
return ['Hello world!\n']

class AppClass:
"""产生同样的输出，不过是使用一个类来实现

(注意: 'AppClass' 在这里就是 "application" ,所以对它的调用会'AppClass'的一个实例,
这个实例做为迭代器再返回"application callable"应该返回的那些值)

如果我们想使用 'AppClass' 的实例直接作为应用程序对象, 我们就必须实现 ``__call__`` 方法,
外部通过调用这个方法来执行应用程序, 并且我们需要创建一个实例给服务器或gateway使用.
"""

def __init__(self, environ, start_response):
self.environ = environ
self.start = start_response

def __iter__(self):
status = '200 OK'
response_headers = [('Content-type','text/plain')]
self.start(status, response_headers)
yield "Hello world!\n"
服务器接口

服务器/gateway为每一个http客户端发来的请求都会请求应用程序可调用者一次。为了说明这里有一个CGI gateway，以一个获取应用程序对象的函数实现，请注意，这个例子拥有有限的错误处理，因为默认情况下没有被捕获的异常都会被输出到sys.stderr并被服务器记录下来。

import os, sys

def run_with_cgi(application):

environ = dict(os.environ.items())
environ['wsgi.input'] = sys.stdin
environ['wsgi.errors'] = sys.stderr
environ['wsgi.version'] = (1,0)
environ['wsgi.multithread'] = False
environ['wsgi.multiprocess'] = True
environ['wsgi.run_once'] = True

if environ.get('HTTPS','off') in ('on','1'):
environ['wsgi.url_scheme'] = 'https'
else:
environ['wsgi.url_scheme'] = 'http'

headers_set = []
headers_sent = []

def write(data):
if not headers_set:
raise AssertionError("write() before start_response()")

elif not headers_sent:
# Before the first output, send the stored headers
status, response_headers = headers_sent[:] = headers_set
sys.stdout.write('Status: %s\r\n' % status)
for header in response_headers:
sys.stdout.write('%s: %s\r\n' % header)
sys.stdout.write('\r\n')

sys.stdout.write(data)
sys.stdout.flush()

def start_response(status,response_headers,exc_info=None):
if exc_info:
try:
if headers_sent:
# Re-raise original exception if headers sent
raise exc_info[0], exc_info[1], exc_info[2]
finally:
exc_info = None # avoid dangling circular ref
elif headers_set:
raise AssertionError("Headers already set!")

headers_set[:] = [status,response_headers]
return write

result = application(environ, start_response)
try:
for data in result:
if data: # body 出现以前不发送headers
write(data)
if not headers_sent:
write('') # 如果这个时候body为空则发送header
finally:
if hasattr(result,'close'):
result.close()
中间件 : 同时扮演两种角色的组件

注意到单个对象可以作为请求应用程序的服务器存在，也可以作为被服务器调用的应用程序存在。这样的中间件可以执行这样一些功能:

重写前面提到的 environ 之后，可以根据目标URL将请求传递到不同的应用程序对象
允许多个应用程序和框架在同一个进程中运行
通过在网络传递请求和响应，实现负载均衡和远程处理
对内容进行后加工，比如附加xsl样式表
中间件的存在对于服务器接口和应用接口来说都应该是透明的，并且不需要特别的支持。希望在应用程序中加入中间件的用户只需简单得把中间件当作应用提供给服务器，并配置中间件足见以服务器的身份来请求应用程序。

当然，中间件组件包裹的可能是包裹应用程序的另一个中间件组件，这样循环下去就构成了我们称为"中间件堆栈"的东西了。 for the most part,中间件要符合应用接口和服务器接口提出的一些限制和要求，有些时候这样的限制甚至比纯粹的服务器或应用程序还要严格，这些地方我们会特别指出。

这里有一个中间件组件的例子，他用Joe Strout的piglatin.py将text/plain的响应转换成pig latin（注意：真正的中间件应该使用更加安全的方式——应该检查内容的类型和内容的编码，同样这个简单的例子还忽略了一个单词might be split across a block boundary的可能性)。

from piglatin import piglatin

class LatinIter:

"""如果可以的话，将输出转换为piglatin格式

Note that the "okayness" can change until the application yields
its first non-empty string, so 'transform_ok' has to be a mutable
truth value."""

def __init__(self,result,transform_ok):
if hasattr(result,'close'):
self.close = result.close
self._next = iter(result).next
self.transform_ok = transform_ok

def __iter__(self):
return self

def next(self):
if self.transform_ok:
return piglatin(self._next())
else:
return self._next()

class Latinator:

# by default, don't transform output
transform = False

def __init__(self, application):
self.application = application

def __call__(environ, start_response):

transform_ok = []

def start_latin(status,response_headers,exc_info=None):

# Reset ok flag, in case this is a repeat call
transform_ok[:]=[]

for name,value in response_headers:
if name.lower()=='content-type' and value=='text/plain':
transform_ok.append(True)
# Strip content-length if present, else it'll be wrong
response_headers = [(name,value)
for name,value in response_headers
if name.lower()<>‘content-length’
]
break

write = start_response(status,response_headers,exc_info)

if transform_ok:
def write_latin(data):
write(piglatin(data))
return write_latin
else:
return write

return LatinIter(self.application(environ,start_latin),transform_ok)

# Run foo_app under a Latinator’s control, using the example CGI gateway
from foo_app import foo_app
run_with_cgi(Latinator(foo_app))
详细说明

应用程序对象必须接受两个参数，为了方便说明我们不妨分别命名为 environ 和 start_response ，但并非必须取这个名字。服务器或gateway必须用这两个参数请求应用程序对象(比如象上面展示的,这样调用 result = application(environ,start_response) )

参数 environ 是个字典对象，包含CGI风格的环境变量。这个对象必须是一个python内建的字典对象(不能是子类、UserDict或其他对字典对象的模仿)，应用程序可以以任何他愿意的方式修改这个字典， environ 还应该包含一些特定的WSGI需要的变量(在后面的节里会描述)，有可以包含一些服务器特定的扩展变量，通过下面提高的约定命名。

start_response 参数是一个接受两个必须参数和一个可选参数的可调用者。方便说明，我们分别把他们命名为 status, response_headers ,和 exc_info 。应用程序必须用这些参数来请求可调用者 start_response (比如象这样 start_response(status,response_headers) )

参数 status 是一个形式象”999 Message here”的状态字符串。而 response_headers 参数是元组(header_name,header_value)的列表,描述http响应头。可选的 exc_info 参数会在下面的 `The start_response() Callable`_ 和 Error Handling 两节中描述，他只有在应用程序产生了错误并希望在浏览器上显示错误的时候才有用。

start_response 可调用者必须返回一个 write(body_data) 可调用者，他接受一个可选参数：一个将要被做为http响应体的一部分输出的字符串(注意：提供可调用者 write() 只是为了支持现有框架的必要的输出API，新的应用程序或框架尽量避免使用，详细情况请看 Buffering and Streaming 一节。)

当被服务器请求的时候，应用程序对象必须返回一个0或多个可迭代的字符串，这可以通过多种方法完成，比如返回一个字符串的列表，或者应用程序本身是一个生产字符串的函数，或者应用程序是一个类而他的实例是可迭代的，不管怎么完成，应用程序对象必须总是返回0或多个可迭代的字符串。

服务器必须将产生的字符串以一种无缓冲的方式传送到客户端，每次传完一个字符串再去获取下一个。(换句话说，应用程序应该实现自己的缓冲，更多关于应用程序输出必须如何处理的细节请阅读下面的 Buffering and Streaming 节。)

服务器或gateway应该把产生的字符串当字节流对待：特别地，他必须保证没修改行的结尾。应用程序负责确保字符串是以与客户端匹配的编码输出(服务器/gateway可能会附加HTTP传送编码，或者为了实现一些http的特性而进行一些转换比如byte-range transmission，更多细节请看下面的 Other HTTP Features )

如果调 len(iterable) 成功，服务器将认为结果是正确的。也就是说，应用程序返回的可迭代的字符串提供了一个有用 的__len__() 方法，么肯定返回了正确的结果(关于这个方法正常情况下如何被使用的请阅读 Handling the Content-Length Header )

如果应用程序返回的可迭代者有close()方法，则不管该请求是正常结束还是由于错误而终止，服务器/gateway都**必须**在结束该请求之前调用这个方法，（这是用来支持应用程序对资源的释放，This protocol is intended to complement PEP 325′s generator support, and other common iterables with close() methods.）

（注意：应用程序必须在可迭代者产生第一个字符串之间请求 start_response() 可调用者，这样服务器才能在发送任何主体内容之前发送响应头，然而这一步也可以在可迭代者第一次迭代的时候执行,所以服务器不能假定开始迭代之前 start_response() 已经被调用过了）

最后，服务器或gateway不能应用程序返回的可迭代者的任何其他属性，除非是针对服务器或gateway特定类型的实例，比如wsgi.file_wrapper返回的“file wrapper”（阅读 Optional Platform-Specific File Handling )。通常情况下，只有在这里指定的属性，或者通过PEP 234 iteration APIs才是可以访问的。

environ 变量

environ 字典被用来包含这些在Common Gateway Interface specification [2]_中定义了的CGI环境变量。 下面这些变量 必须 呈现出来, 除非其值是空字符串,这种情况下如果下面没有特别指出的话他们 可能 会被忽略

REQUEST_METHOD
HTTP请求的方式, 比如 “GET” 或者 “POST”. 这个不可能是空字符串并且也是必须给出的。
SCRIPT_NAME
请求URL中路径的开始部分，对应应用程序对象，这样应用程序就知道它的虚拟位置。 如果该应用程序对应服务器的 根 的话， 它 可能 是为空字符串。
PATH_INFO
请求URL中路径的剩余部分，指定请求的目标在应用程序内部的虚拟位置。 如果请求的目标是应用程序跟并且没有trailing slash的话，可能为空字符串 。
QUERY_STRING
请求URL中跟在”?”后面的那部分,可能为空或不存在.
CONTENT_TYPE
HTTP请求中任何 Content-Type 域的内容。
CONTENT_LENGTH
HTTP请求中任何 Content-Length 域的内容。 可能为空或不存在.
SERVER_NAME, SERVER_PORT
这些变量可以和 SCRIPT_NAME、PATH_INFO 一起组成完整的URL。 然而要注意的是，重建请求URL的时候应该优先使用 HTTP_HOST 而非 SERVER_NAME 。 详细内容请阅读下面的 URL Reconstruction 。 SERVER_NAME 和 SERVER_PORT 永远 是空字符串，也总是必须存在的。
SERVER_PROTOCOL
客户端发送请求所使用协议的版本。通常是类似 “HTTP/1.0″ 或 “HTTP/1.1″ 的东西 可以被用来判断如何处理请求headers。(既然这个变量表示的是请求中使用的协议，而且和服 务器响应时使用的协议无关，也许它应该被叫做 REQUEST_PROTOCOL 。然后，为了保持和 CGI的兼容性，我们还是使用已有的名字。)
HTTP_ 变量
对应客户端提供的HTTP请求headers (也就是说名字以 “HTTP_” 开头的变量)。这些变量的 存在与否应该和请求中的合适的HTTP header一致。
服务器或gateway 应该 尽可能提供其他可用的CGI变量。另外，如果用了SSL，服务器或 gateway也 应该 尽可能提供可用的Apache SSL环境变量 [5] ，比如 HTTPS=on 和 SSL_PROTOCOL“。不过要注意，使用了任何上面没有列出的变量的应用程序对不支持相关扩展 的服务器来说就有点necessarily non-portable。(比如，不发布文件的web服务器就不能提供一 个有意义的 “DOCUMENT_ROOT 或 PATH_TRANSLATED 。)

一个支持WSGI的服务器或gateway 应该 在描述它们自己的同时说明它们可以提供些什么变量 应用程序 应该 对所有他们需要的变量的存在性进行检查，并且在某变量不存在的时候有 备用的措施

注意: 不需要的变量 (比如在不需要验证的情况下的 REMOTE_USER ) 应该被移出 environ 字典。同样注意CGI定义的变量如果存在的话必须是字符串。任何 str 类型以外的CGI变量的 存在都是对本规范的违反

除了CGI定义的变量， environ 字典也可以包含任意操作系统的环境变量，并且 必须 包含下面这些WSGI定义的变量:

变量 值
wsgi.version 元组 (1,0), 表明WSGI 版本 1.0.
wsgi.url_scheme A string representing the “scheme” portion of the URL at which the application is being invoked. Normally, this will have the value “http” or “https”, as appropriate.
wsgi.input An input stream (file-like object) from which the HTTP request body can be read. (The server or gateway may perform reads on-demand as requested by the application, or it may pre- read the client’s request body and buffer it in-memory or on disk, or use any other technique for providing such an input stream, according to its preference.)
wsgi.errors
An output stream (file-like object) to which error output can be written, for the purpose of recording program or other errors in a standardized and possibly centralized location. This should be a “text mode” stream; i.e., applications should use “\n” as a line ending, and assume that it will be converted to the correct line ending by the server/gateway.

For many servers, wsgi.errors will be the server’s main error log. Alternatively, this may be sys.stderr, or a log file of some sort. The server’s documentation should include an explanation of how to configure this or where to find the recorded output. A server or gateway may supply different error streams to different applications, if this is desired.
wsgi.multithread This value should evaluate true if the application object may be simultaneously invoked by another thread in the same process, and should evaluate false otherwise.
wsgi.multiprocess This value should evaluate true if an equivalent application object may be simultaneously invoked by another process, and should evaluate false otherwise.
wsgi.run_once This value should evaluate true if the server or gateway expects (but does not guarantee!) that the application will only be invoked this one time during the life of its containing process. Normally, this will only be true for a gateway based on CGI (or something similar).
最后 environ 字典也可以包含服务器定义的变量。这些变量的名字必须是小写字母、数字、点和下划线， 并且应该带一个能唯一代表服务器或gateway的前缀。比如， mod_python 可能会定义象这样的一些变量: mod_python.some_variable.

输入和错误流

服务器提供的输入和错误流必须提供以下方法:

方法 流 注解
read(size) input 1
readline() input 1,2
readlines(hint) input 1,3
__iter__() input
flush() errors 4
write(str) errors
writelines(seq) errors
每个方法的语义如果上面没有特别指出均和Python Library Reference记载的一样:

The server is not required to read past the client’s specified Content-Length, and is allowed to simulate an end-of-file condition if the application attempts to read past that point. The application should not attempt to read more data than is specified by the CONTENT_LENGTH variable.
The optional “size” argument to readline() is not supported, as it may be complex for server authors to implement, and is not often used in practice.
Note that the hint argument to readlines() is optional for both caller and implementer. The application is free not to supply it, and the server or gateway is free to ignore it.
Since the errors stream may not be rewound, servers and gateways are free to forward write operations immediately, without buffering. In this case, the flush() method may be a no-op. Portable applications, however, cannot assume that output is unbuffered or that flush() is a no-op. They must call flush() if they need to ensure that output has in fact been written. (For example, to minimize intermingling of data from multiple processes writing to the same error log.)
The methods listed in the table above must be supported by all servers conforming to this specification. Applications conforming to this specification must not use any other methods or attributes of the input or errors objects. In particular, applications must not attempt to close these streams, even if they possess close() methods.

start_response() 可调用者

传给应用程序对象的第二个参数是一个形为 start_response(status,response_headers,exc_info=None) 的可调用者. (As with all WSGI callables, the arguments must be supplied positionally, not by keyword.) start_response 可调用者是用来开始HTTP响应，它必须返回一个 write(body_data) 可调用者 (阅读下面的 Buffering and Streaming).

status“参数是一个HTTP “status” 字符串，比如 “”200 OK” 或 “404 Not Found”. 也就是说，他是一个由状态编号和具体信息组成的字符串，按这个顺序并用空格隔开，两头没有其他空格和其他字符。 (更多信息请阅读RFC 2616, Section 6.1.1) 该字符串 禁止 包含控制字符，也不允许以回车、换行或他们的 组合结束。

response_headers“参数是一个 “(header_name,header_value) 元组的列表。它必须是一个Python列表； 也就是说 type(response_headers) is ListType,并且服务器 可以 以任何方式改变其内容。 每一个 header_name 必须是一个没有冒号或其他标点符号的合法的HTTP header字段名 (在RFC 2616, Section 4.2中有详细定义).

每一个 header_value 禁止 包含 任何 控制字符,包括回车或换行。 (这些要求是要使得那些必须检查或修改响应头的服务器、gateway、响应处理中间件所必须执行的解析工作的 复杂性降到最低。)

In general, the server or gateway is responsible for ensuring that correct headers are sent to the client: if the application omits a header required by HTTP (or other relevant specifications that are in effect), the server or gateway must add it. For example, the HTTP Date: and Server: headers would normally be supplied by the server or gateway.

(A reminder for server/gateway authors: HTTP header names are case-insensitive, so be sure to take that into consideration when examining application-supplied headers!)

Applications and middleware are forbidden from using HTTP/1.1 “hop-by-hop” features or headers, any equivalent features in HTTP/1.0, or any headers that would affect the persistence of the client’s connection to the web server. These features are the exclusive province of the actual web server, and a server or gateway should consider it a fatal error for an application to attempt sending them, and raise an error if they are supplied to start_response(). (For more specifics on “hop-by-hop” features and headers, please see the Other HTTP Features section below.)

The start_response callable must not actually transmit the response headers. Instead, it must store them for the server or gateway to transmit only after the first iteration of the application return value that yields a non-empty string, or upon the application’s first invocation of the write() callable. In other words, response headers must not be sent until there is actual body data available, or until the application’s returned iterable is exhausted. (The only possible exception to this rule is if the response headers explicitly include a Content-Length of zero.)

This delaying of response header transmission is to ensure that buffered and asynchronous applications can replace their originally intended output with error output, up until the last possible moment. For example, the application may need to change the response status from “200 OK” to “500 Internal Error”, if an error occurs while the body is being generated within an application buffer.

The exc_info argument, if supplied, must be a Python sys.exc_info() tuple. This argument should be supplied by the application only if start_response is being called by an error handler. If exc_info is supplied, and no HTTP headers have been output yet, start_response should replace the currently-stored HTTP response headers with the newly-supplied ones, thus allowing the application to “change its mind” about the output when an error has occurred.

However, if exc_info is provided, and the HTTP headers have already been sent, start_response must raise an error, and should raise the exc_info tuple. That is:

raise exc_info[0],exc_info[1],exc_info[2]
This will re-raise the exception trapped by the application, and in principle should abort the application. (It is not safe for the application to attempt error output to the browser once the HTTP headers have already been sent.) The application must not trap any exceptions raised by start_response, if it called start_response with exc_info. Instead, it should allow such exceptions to propagate back to the server or gateway. See Error Handling below, for more details.

The application may call start_response more than once, if and only if the exc_info argument is provided. More precisely, it is a fatal error to call start_response without the exc_info argument if start_response has already been called within the current invocation of the application. (See the example CGI gateway above for an illustration of the correct logic.)

Note: servers, gateways, or middleware implementing start_response should ensure that no reference is held to the exc_info parameter beyond the duration of the function’s execution, to avoid creating a circular reference through the traceback and frames involved. The simplest way to do this is something like:

def start_response(status,response_headers,exc_info=None):
if exc_info:
try:
# do stuff w/exc_info here
finally:
exc_info = None # Avoid circular ref.
The example CGI gateway provides another illustration of this technique.

Handling the Content-Length Header

If the application does not supply a Content-Length header, a server or gateway may choose one of several approaches to handling it. The simplest of these is to close the client connection when the response is completed.

Under some circumstances, however, the server or gateway may be able to either generate a Content-Length header, or at least avoid the need to close the client connection. If the application does not call the write() callable, and returns an iterable whose len() is 1, then the server can automatically determine Content-Length by taking the length of the first string yielded by the iterable.

And, if the server and client both support HTTP/1.1 “chunked encoding” [3], then the server may use chunked encoding to send a chunk for each write() call or string yielded by the iterable, thus generating a Content-Length header for each chunk. This allows the server to keep the client connection alive, if it wishes to do so. Note that the server must comply fully with RFC 2616 when doing this, or else fall back to one of the other strategies for dealing with the absence of Content-Length.

(Note: applications and middleware must not apply any kind of Transfer-Encoding to their output, such as chunking or gzipping; as “hop-by-hop” operations, these encodings are the province of the actual web server/gateway. See Other HTTP Features below, for more details.)

Buffering and Streaming

Generally speaking, applications will achieve the best throughput by buffering their (modestly-sized) output and sending it all at once. This is a common approach in existing frameworks such as Zope: the output is buffered in a StringIO or similar object, then transmitted all at once, along with the response headers.

The corresponding approach in WSGI is for the application to simply return a single-element iterable (such as a list) containing the response body as a single string. This is the recommended approach for the vast majority of application functions, that render HTML pages whose text easily fits in memory.

For large files, however, or for specialized uses of HTTP streaming (such as multipart “server push”), an application may need to provide output in smaller blocks (e.g. to avoid loading a large file into memory). It’s also sometimes the case that part of a response may be time-consuming to produce, but it would be useful to send ahead the portion of the response that precedes it.

In these cases, applications will usually return an iterator (often a generator-iterator) that produces the output in a block-by-block fashion. These blocks may be broken to coincide with mulitpart boundaries (for “server push”), or just before time-consuming tasks (such as reading another block of an on-disk file).

WSGI servers, gateways, and middleware must not delay the transmission of any block; they must either fully transmit the block to the client, or guarantee that they will continue transmission even while the application is producing its next block. A server/gateway or middleware may provide this guarantee in one of three ways:

Send the entire block to the operating system (and request that any O/S buffers be flushed) before returning control to the application, OR
Use a different thread to ensure that the block continues to be transmitted while the application produces the next block.
(Middleware only) send the entire block to its parent gateway/server
By providing this guarantee, WSGI allows applications to ensure that transmission will not become stalled at an arbitrary point in their output data. This is critical for proper functioning of e.g. multipart “server push” streaming, where data between multipart boundaries should be transmitted in full to the client.

Middleware Handling of Block Boundaries

In order to better support asynchronous applications and servers, middleware components must not block iteration waiting for multiple values from an application iterable. If the middleware needs to accumulate more data from the application before it can produce any output, it must yield an empty string.

To put this requirement another way, a middleware component must yield at least one value each time its underlying application yields a value. If the middleware cannot yield any other value, it must yield an empty string.

This requirement ensures that asynchronous applications and servers can conspire to reduce the number of threads that are required to run a given number of application instances simultaneously.

Note also that this requirement means that middleware must return an iterable as soon as its underlying application returns an iterable. It is also forbidden for middleware to use the write() callable to transmit data that is yielded by an underlying application. Middleware may only use their parent server’s write() callable to transmit data that the underlying application sent using a middleware-provided write() callable.

The write() Callable

Some existing application framework APIs support unbuffered output in a different manner than WSGI. Specifically, they provide a “write” function or method of some kind to write an unbuffered block of data, or else they provide a buffered “write” function and a “flush” mechanism to flush the buffer.

Unfortunately, such APIs cannot be implemented in terms of WSGI’s “iterable” application return value, unless threads or other special mechanisms are used.

Therefore, to allow these frameworks to continue using an imperative API, WSGI includes a special write() callable, returned by the start_response callable.

New WSGI applications and frameworks should not use the write() callable if it is possible to avoid doing so. The write() callable is strictly a hack to support imperative streaming APIs. In general, applications should produce their output via their returned iterable, as this makes it possible for web servers to interleave other tasks in the same Python thread, potentially providing better throughput for the server as a whole.

The write() callable is returned by the start_response() callable, and it accepts a single parameter: a string to be written as part of the HTTP response body, that is treated exactly as though it had been yielded by the output iterable. In other words, before write() returns, it must guarantee that the passed-in string was either completely sent to the client, or that it is buffered for transmission while the application proceeds onward.

An application must return an iterable object, even if it uses write() to produce all or part of its response body. The returned iterable may be empty (i.e. yield no non-empty strings), but if it does yield non-empty strings, that output must be treated normally by the server or gateway (i.e., it must be sent or queued immediately). Applications must not invoke write() from within their return iterable, and therefore any strings yielded by the iterable are transmitted after all strings passed to write() have been sent to the client.

Unicode Issues

HTTP does not directly support Unicode, and neither does this interface. All encoding/decoding must be handled by the application; all strings passed to or from the server must be standard Python byte strings, not Unicode objects. The result of using a Unicode object where a string object is required, is undefined.

Note also that strings passed to start_response() as a status or as response headers must follow RFC 2616 with respect to encoding. That is, they must either be ISO-8859-1 characters, or use RFC 2047 MIME encoding.

On Python platforms where the str or StringType type is in fact Unicode-based (e.g. Jython, IronPython, Python 3000, etc.), all “strings” referred to in this specification must contain only code points representable in ISO-8859-1 encoding (\u0000 through \u00FF, inclusive). It is a fatal error for an application to supply strings containing any other Unicode character or code point. Similarly, servers and gateways must not supply strings to an application containing any other Unicode characters.

Again, all strings referred to in this specification must be of type str or StringType, and must not be of type unicode or UnicodeType. And, even if a given platform allows for more than 8 bits per character in str/StringType objects, only the lower 8 bits may be used, for any value referred to in this specification as a “string”.

Error Handling

In general, applications should try to trap their own, internal errors, and display a helpful message in the browser. (It is up to the application to decide what “helpful” means in this context.)

However, to display such a message, the application must not have actually sent any data to the browser yet, or else it risks corrupting the response. WSGI therefore provides a mechanism to either allow the application to send its error message, or be automatically aborted: the exc_info argument to start_response. Here is an example of its use:

try:
# regular application code here
status = “200 Froody”
response_headers = [("content-type","text/plain")]
start_response(status, response_headers)
return ["normal body goes here"]
except:
# XXX should trap runtime issues like MemoryError, KeyboardInterrupt
# in a separate handler before this bare ‘except:’…
status = “500 Oops”
response_headers = [("content-type","text/plain")]
start_response(status, response_headers, sys.exc_info())
return ["error body goes here"]
If no output has been written when an exception occurs, the call to start_response will return normally, and the application will return an error body to be sent to the browser. However, if any output has already been sent to the browser, start_response will reraise the provided exception. This exception should not be trapped by the application, and so the application will abort. The server or gateway can then trap this (fatal) exception and abort the response.

Servers should trap and log any exception that aborts an application or the iteration of its return value. If a partial response has already been written to the browser when an application error occurs, the server or gateway may attempt to add an error message to the output, if the already-sent headers indicate a text/* content type that the server knows how to modify cleanly.

Some middleware may wish to provide additional exception handling services, or intercept and replace application error messages. In such cases, middleware may choose to not re-raise the exc_info supplied to start_response, but instead raise a middleware-specific exception, or simply return without an exception after storing the supplied arguments. This will then cause the application to return its error body iterable (or invoke write()), allowing the middleware to capture and modify the error output. These techniques will work as long as application authors:

Always provide exc_info when beginning an error response
Never trap errors raised by start_response when exc_info is being provided
HTTP 1.1 Expect/Continue

Servers and gateways that implement HTTP 1.1 must provide transparent support for HTTP 1.1′s “expect/continue” mechanism. This may be done in any of several ways:

Respond to requests containing an Expect: 100-continue request with an immediate “100 Continue” response, and proceed normally.
Proceed with the request normally, but provide the application with a wsgi.input stream that will send the “100 Continue” response if/when the application first attempts to read from the input stream. The read request must then remain blocked until the client responds.
Wait until the client decides that the server does not support expect/continue, and sends the request body on its own. (This is suboptimal, and is not recommended.)
Note that these behavior restrictions do not apply for HTTP 1.0 requests, or for requests that are not directed to an application object. For more information on HTTP 1.1 Expect/Continue, see RFC 2616, sections 8.2.3 and 10.1.1.

Other HTTP Features

In general, servers and gateways should “play dumb” and allow the application complete control over its output. They should only make changes that do not alter the effective semantics of the application’s response. It is always possible for the application developer to add middleware components to supply additional features, so server/gateway developers should be conservative in their implementation. In a sense, a server should consider itself to be like an HTTP “gateway server”, with the application being an HTTP “origin server”. (See RFC 2616, section 1.3, for the definition of these terms.)

However, because WSGI servers and applications do not communicate via HTTP, what RFC 2616 calls “hop-by-hop” headers do not apply to WSGI internal communications. WSGI applications must not generate any “hop-by-hop” headers [4], attempt to use HTTP features that would require them to generate such headers, or rely on the content of any incoming “hop-by-hop” headers in the environ dictionary. WSGI servers must handle any supported inbound “hop-by-hop” headers on their own, such as by decoding any inbound Transfer-Encoding, including chunked encoding if applicable.

Applying these principles to a variety of HTTP features, it should be clear that a server may handle cache validation via the If-None-Match and If-Modified-Since request headers and the Last-Modified and ETag response headers. However, it is not required to do this, and the application should perform its own cache validation if it wants to support that feature, since the server/gateway is not required to do such validation.

Similarly, a server may re-encode or transport-encode an application’s response, but the application should use a suitable content encoding on its own, and must not apply a transport encoding. A server may transmit byte ranges of the application’s response if requested by the client, and the application doesn’t natively support byte ranges. Again, however, the application should perform this function on its own if desired.

Note that these restrictions on applications do not necessarily mean that every application must reimplement every HTTP feature; many HTTP features can be partially or fully implemented by middleware components, thus freeing both server and application authors from implementing the same features over and over again.

Thread Support

Thread support, or lack thereof, is also server-dependent. Servers that can run multiple requests in parallel, should also provide the option of running an application in a single-threaded fashion, so that applications or frameworks that are not thread-safe may still be used with that server.

Implementation/Application Notes

Server Extension APIs

Some server authors may wish to expose more advanced APIs, that application or framework authors can use for specialized purposes. For example, a gateway based on mod_python might wish to expose part of the Apache API as a WSGI extension.

In the simplest case, this requires nothing more than defining an environ variable, such as mod_python.some_api. But, in many cases, the possible presence of middleware can make this difficult. For example, an API that offers access to the same HTTP headers that are found in environ variables, might return different data if environ has been modified by middleware.

In general, any extension API that duplicates, supplants, or bypasses some portion of WSGI functionality runs the risk of being incompatible with middleware components. Server/gateway developers should not assume that nobody will use middleware, because some framework developers specifically intend to organize or reorganize their frameworks to function almost entirely as middleware of various kinds.

So, to provide maximum compatibility, servers and gateways that provide extension APIs that replace some WSGI functionality, must design those APIs so that they are invoked using the portion of the API that they replace. For example, an extension API to access HTTP request headers must require the application to pass in its current environ, so that the server/gateway may verify that HTTP headers accessible via the API have not been altered by middleware. If the extension API cannot guarantee that it will always agree with environ about the contents of HTTP headers, it must refuse service to the application, e.g. by raising an error, returning None instead of a header collection, or whatever is appropriate to the API.

Similarly, if an extension API provides an alternate means of writing response data or headers, it should require the start_response callable to be passed in, before the application can obtain the extended service. If the object passed in is not the same one that the server/gateway originally supplied to the application, it cannot guarantee correct operation and must refuse to provide the extended service to the application.

These guidelines also apply to middleware that adds information such as parsed cookies, form variables, sessions, and the like to environ. Specifically, such middleware should provide these features as functions which operate on environ, rather than simply stuffing values into environ. This helps ensure that information is calculated from environ after any middleware has done any URL rewrites or other environ modifications.

It is very important that these “safe extension” rules be followed by both server/gateway and middleware developers, in order to avoid a future in which middleware developers are forced to delete any and all extension APIs from environ to ensure that their mediation isn’t being bypassed by applications using those extensions!

Application Configuration

This specification does not define how a server selects or obtains an application to invoke. These and other configuration options are highly server-specific matters. It is expected that server/gateway authors will document how to configure the server to execute a particular application object, and with what options (such as threading options).

Framework authors, on the other hand, should document how to create an application object that wraps their framework’s functionality. The user, who has chosen both the server and the application framework, must connect the two together. However, since both the framework and the server now have a common interface, this should be merely a mechanical matter, rather than a significant engineering effort for each new server/framework pair.

Finally, some applications, frameworks, and middleware may wish to use the environ dictionary to receive simple string configuration options. Servers and gateways should support this by allowing an application’s deployer to specify name-value pairs to be placed in environ. In the simplest case, this support can consist merely of copying all operating system-supplied environment variables from os.environ into the environ dictionary, since the deployer in principle can configure these externally to the server, or in the CGI case they may be able to be set via the server’s configuration files.

Applications should try to keep such required variables to a minimum, since not all servers will support easy configuration of them. Of course, even in the worst case, persons deploying an application can create a script to supply the necessary configuration values:

from the_app import application

def new_app(environ,start_response):
environ['the_app.configval1'] = ‘something’
return application(environ,start_response)
But, most existing applications and frameworks will probably only need a single configuration value from environ, to indicate the location of their application or framework-specific configuration file(s). (Of course, applications should cache such configuration, to avoid having to re-read it upon each invocation.)

URL Reconstruction

If an application wishes to reconstruct a request’s complete URL, it may do so using the following algorithm, contributed by Ian Bicking:

from urllib import quote
url = environ['wsgi.url_scheme']+’://’

if environ.get(‘HTTP_HOST’):
url += environ['HTTP_HOST']
else:
url += environ['SERVER_NAME']

if environ['wsgi.url_scheme'] == ‘https’:
if environ['SERVER_PORT'] != ’443′:
url += ‘:’ + environ['SERVER_PORT']
else:
if environ['SERVER_PORT'] != ’80′:
url += ‘:’ + environ['SERVER_PORT']

url += quote(environ.get(‘SCRIPT_NAME’,”))
url += quote(environ.get(‘PATH_INFO’,”))
if environ.get(‘QUERY_STRING’):
url += ‘?’ + environ['QUERY_STRING']
Note that such a reconstructed URL may not be precisely the same URI as requested by the client. Server rewrite rules, for example, may have modified the client’s originally requested URL to place it in a canonical form.

Supporting Older (<2.2) Versions of Python

Some servers, gateways, or applications may wish to support older (<2.2) versions of Python. This is especially important if Jython is a target platform, since as of this writing a production-ready version of Jython 2.2 is not yet available.

For servers and gateways, this is relatively straightforward: servers and gateways targeting pre-2.2 versions of Python must simply restrict themselves to using only a standard “for” loop to iterate over any iterable returned by an application. This is the only way to ensure source-level compatibility with both the pre-2.2 iterator protocol (discussed further below) and “today’s” iterator protocol (see PEP 234).

(Note that this technique necessarily applies only to servers, gateways, or middleware that are written in Python. Discussion of how to use iterator protocol(s) correctly from other languages is outside the scope of this PEP.)

For applications, supporting pre-2.2 versions of Python is slightly more complex:

You may not return a file object and expect it to work as an iterable, since before Python 2.2, files were not iterable. (In general, you shouldn’t do this anyway, because it will peform quite poorly most of the time!) Use wsgi.file_wrapper or an application-specific file wrapper class. (See Optional Platform-Specific File Handling for more on wsgi.file_wrapper, and an example class you can use to wrap a file as an iterable.)
If you return a custom iterable, it must implement the pre-2.2 iterator protocol. That is, provide a __getitem__ method that accepts an integer key, and raises IndexError when exhausted. (Note that built-in sequence types are also acceptable, since they also implement this protocol.)
Finally, middleware that wishes to support pre-2.2 versions of Python, and iterates over application return values or itself returns an iterable (or both), must follow the appropriate recommendations above.

(Note: It should go without saying that to support pre-2.2 versions of Python, any server, gateway, application, or middleware must also use only language features available in the target version, use 1 and 0 instead of True and False, etc.)

Optional Platform-Specific File Handling

Some operating environments provide special high-performance file- transmission facilities, such as the Unix sendfile() call. Servers and gateways may expose this functionality via an optional wsgi.file_wrapper key in the environ. An application may use this “file wrapper” to convert a file or file-like object into an iterable that it then returns, e.g.:

if ‘wsgi.file_wrapper’ in environ:
return environ['wsgi.file_wrapper'](filelike, block_size)
else:
return iter(lambda: filelike.read(block_size), ”)
If the server or gateway supplies wsgi.file_wrapper, it must be a callable that accepts one required positional parameter, and one optional positional parameter. The first parameter is the file-like object to be sent, and the second parameter is an optional block size “suggestion” (which the server/gateway need not use). The callable must return an iterable object, and must not perform any data transmission until and unless the server/gateway actually receives the iterable as a return value from the application. (To do otherwise would prevent middleware from being able to interpret or override the response data.)

To be considered “file-like”, the object supplied by the application must have a read() method that takes an optional size argument. It may have a close() method, and if so, the iterable returned by wsgi.file_wrapper must have a close() method that invokes the original file-like object’s close() method. If the “file-like” object has any other methods or attributes with names matching those of Python built-in file objects (e.g. fileno()), the wsgi.file_wrapper may assume that these methods or attributes have the same semantics as those of a built-in file object.

The actual implementation of any platform-specific file handling must occur after the application returns, and the server or gateway checks to see if a wrapper object was returned. (Again, because of the presence of middleware, error handlers, and the like, it is not guaranteed that any wrapper created will actually be used.)

Apart from the handling of close(), the semantics of returning a file wrapper from the application should be the same as if the application had returned iter(filelike.read, ”). In other words, transmission should begin at the current position within the “file” at the time that transmission begins, and continue until the end is reached.

Of course, platform-specific file transmission APIs don’t usually accept arbitrary “file-like” objects. Therefore, a wsgi.file_wrapper has to introspect the supplied object for things such as a fileno() (Unix-like OSes) or a java.nio.FileChannel (under Jython) in order to determine if the file-like object is suitable for use with the platform-specific API it supports.

Note that even if the object is not suitable for the platform API, the wsgi.file_wrapper must still return an iterable that wraps read() and close(), so that applications using file wrappers are portable across platforms. Here’s a simple platform-agnostic file wrapper class, suitable for old (pre 2.2) and new Pythons alike:

class FileWrapper:

def __init__(self, filelike, blksize=8192):
self.filelike = filelike
self.blksize = blksize
if hasattr(filelike,’close’):
self.close = filelike.close

def __getitem__(self,key):
data = self.filelike.read(self.blksize)
if data:
return data
raise IndexError
and here is a snippet from a server/gateway that uses it to provide access to a platform-specific API:

environ['wsgi.file_wrapper'] = FileWrapper
result = application(environ, start_response)

try:
if isinstance(result,FileWrapper):
# check if result.filelike is usable w/platform-specific
# API, and if so, use that API to transmit the result.
# If not, fall through to normal iterable handling
# loop below.

for data in result:
# etc.

finally:
if hasattr(result,’close’):
result.close()
Questions and Answers

Why must environ be a dictionary? What’s wrong with using a subclass?

The rationale for requiring a dictionary is to maximize portability between servers. The alternative would be to define some subset of a dictionary’s methods as being the standard and portable interface. In practice, however, most servers will probably find a dictionary adequate to their needs, and thus framework authors will come to expect the full set of dictionary features to be available, since they will be there more often than not. But, if some server chooses not to use a dictionary, then there will be interoperability problems despite that server’s “conformance” to spec. Therefore, making a dictionary mandatory simplifies the specification and guarantees interoperabilty.

Note that this does not prevent server or framework developers from offering specialized services as custom variables inside the environ dictionary. This is the recommended approach for offering any such value-added services.

Why can you call write() and yield strings/return an iterable? Shouldn’t we pick just one way?

If we supported only the iteration approach, then current frameworks that assume the availability of “push” suffer. But, if we only support pushing via write(), then server performance suffers for transmission of e.g. large files (if a worker thread can’t begin work on a new request until all of the output has been sent). Thus, this compromise allows an application framework to support both approaches, as appropriate, but with only a little more burden to the server implementor than a push-only approach would require.

What’s the close() for?

When writes are done during the execution of an application object, the application can ensure that resources are released using a try/finally block. But, if the application returns an iterable, any resources used will not be released until the iterable is garbage collected. The close() idiom allows an application to release critical resources at the end of a request, and it’s forward-compatible with the support for try/finally in generators that’s proposed by PEP 325.

Why is this interface so low-level? I want feature X! (e.g. cookies, sessions, persistence, …)

This isn’t Yet Another Python Web Framework. It’s just a way for frameworks to talk to web servers, and vice versa. If you want these features, you need to pick a web framework that provides the features you want. And if that framework lets you create a WSGI application, you should be able to run it in most WSGI-supporting servers. Also, some WSGI servers may offer additional services via objects provided in their environ dictionary; see the applicable server documentation for details. (Of course, applications that use such extensions will not be portable to other WSGI-based servers.)

Why use CGI variables instead of good old HTTP headers? And why mix them in with WSGI-defined variables?

Many existing web frameworks are built heavily upon the CGI spec, and existing web servers know how to generate CGI variables. In contrast, alternative ways of representing inbound HTTP information are fragmented and lack market share. Thus, using the CGI “standard” seems like a good way to leverage existing implementations. As for mixing them with WSGI variables, separating them would just require two dictionary arguments to be passed around, while providing no real benefits.

What about the status string? Can’t we just use the number, passing in 200 instead of “200 OK”?

Doing this would complicate the server or gateway, by requiring them to have a table of numeric statuses and corresponding messages. By contrast, it is easy for an application or framework author to type the extra text to go with the specific response code they are using, and existing frameworks often already have a table containing the needed messages. So, on balance it seems better to make the application/framework responsible, rather than the server or gateway.

Why is wsgi.run_once not guaranteed to run the app only once?

Because it’s merely a suggestion to the application that it should “rig for infrequent running”. This is intended for application frameworks that have multiple modes of operation for caching, sessions, and so forth. In a “multiple run” mode, such frameworks may preload caches, and may not write e.g. logs or session data to disk after each request. In “single run” mode, such frameworks avoid preloading and flush all necessary writes after each request.

However, in order to test an application or framework to verify correct operation in the latter mode, it may be necessary (or at least expedient) to invoke it more than once. Therefore, an application should not assume that it will definitely not be run again, just because it is called with wsgi.run_once set to True.

Feature X (dictionaries, callables, etc.) are ugly for use in application code; why don’t we use objects instead?

All of these implementation choices of WSGI are specifically intended to decouple features from one another; recombining these features into encapsulated objects makes it somewhat harder to write servers or gateways, and an order of magnitude harder to write middleware that replaces or modifies only small portions of the overall functionality.

In essence, middleware wants to have a “Chain of Responsibility” pattern, whereby it can act as a “handler” for some functions, while allowing others to remain unchanged. This is difficult to do with ordinary Python objects, if the interface is to remain extensible. For example, one must use __getattr__ or __getattribute__ overrides, to ensure that extensions (such as attributes defined by future WSGI versions) are passed through.

This type of code is notoriously difficult to get 100% correct, and few people will want to write it themselves. They will therefore copy other people’s implementations, but fail to update them when the person they copied from corrects yet another corner case.

Further, this necessary boilerplate would be pure excise, a developer tax paid by middleware developers to support a slightly prettier API for application framework developers. But, application framework developers will typically only be updating one framework to support WSGI, and in a very limited part of their framework as a whole. It will likely be their first (and maybe their only) WSGI implementation, and thus they will likely implement with this specification ready to hand. Thus, the effort of making the API “prettier” with object attributes and suchlike would likely be wasted for this audience.

We encourage those who want a prettier (or otherwise improved) WSGI interface for use in direct web application programming (as opposed to web framework development) to develop APIs or frameworks that wrap WSGI for convenient use by application developers. In this way, WSGI can remain conveniently low-level for server and middleware authors, while not being “ugly” for application developers.

Proposed/Under Discussion

These items are currently being discussed on the Web-SIG and elsewhere, or are on the PEP author’s “to-do” list:

Should wsgi.input be an iterator instead of a file? This would help for asynchronous applications and chunked-encoding input streams.
Optional extensions are being discussed for pausing iteration of an application’s ouptut until input is available or until a callback occurs.
Add a section about synchronous vs. asynchronous apps and servers, the relevant threading models, and issues/design goals in these areas.
Acknowledgements

Thanks go to the many folks on the Web-SIG mailing list whose thoughtful feedback made this revised draft possible. Especially:

Gregory “Grisha” Trubetskoy, author of mod_python, who beat up on the first draft as not offering any advantages over “plain old CGI”, thus encouraging me to look for a better approach.
Ian Bicking, who helped nag me into properly specifying the multithreading and multiprocess options, as well as badgering me to provide a mechanism for servers to supply custom extension data to an application.
Tony Lownds, who came up with the concept of a start_response function that took the status and headers, returning a write function. His input also guided the design of the exception handling facilities, especially in the area of allowing for middleware that overrides application error messages.
Alan Kennedy, whose courageous attempts to implement WSGI-on-Jython (well before the spec was finalized) helped to shape the “supporting older versions of Python” section, as well as the optional wsgi.file_wrapper facility.
Mark Nottingham, who reviewed the spec extensively for issues with HTTP RFC compliance, especially with regard to HTTP/1.1 features that I didn’t even know existed until he pointed them out.
References

[1] The Python Wiki “Web Programming” topic (http://www.python.org/cgi-bin/moinmoin/WebProgramming)
[2] The Common Gateway Interface Specification, v 1.1, 3rd Draft (http://cgi-spec.golux.com/draft-coar-cgi-v11-03.txt)
[3] “Chunked Transfer Coding” — HTTP/1.1, section 3.6.1 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1)
[4] “End-to-end and Hop-by-hop Headers” — HTTP/1.1, Section 13.5.1 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.5.1)
[5] mod_ssl Reference, “Environment Variables” (http://www.modssl.org/docs/2.8/ssl_reference.html#ToC25)
Copyright

This document has been placed in the public domain.

Docutils System Messages

System Message: ERROR/3 (pep-0333_cn_temp.txt, line 259); backlink

Unknown target name: “the start_response() callable”.

您可能也喜欢：
超级简单Python Socket Server一例	Python PIL 生成带阴影的缩略图	python非贪婪、多行匹配正则表达式例子	Python Web.py框架实现的简易REST服务原型	Python 批量生成缩略图
无觅

(SELECT a FROM tbl_name WHERE a=10 AND B=1)  
UNION  
(SELECT a FROM tbl_name WHERE a=11 AND B=2)  
ORDER BY a LIMIT 10；

如果，需要对单个SQL语句进行order by，则应把order by子句放入圆括号中，如下：

(SELECT a FROM tbl_name WHERE a=10 AND B=1 ORDER BY a LIMIT 10)  
UNION  
(SELECT a FROM tbl_name WHERE a=11 AND B=2 ORDER BY a LIMIT 10);

另外注意：圆括号中用于单个SQL语句的ORDER BY只有当与LIMIT结合后，才起作用。否则，ORDER BY被优化去除。
我在innodb引擎的2个表上测试了没有加()，进行union和order by的操作，如下：

SELECT a FROM tbl_name WHERE a=10 AND B=1  
UNION  
SELECT a FROM tbl_name WHERE a=11 AND B=2  
ORDER BY a LIMIT 10;

发现，它默认的结果也是先进行union，然后再order by，和第一种情况执行结果相同
不过，为了逻辑清晰，最好还是加上对应的()比较好
另外：Mysql中union可以有union，union distinct，union all这3中形式
union和union distinct会对union后的结果进行排重，保证所有返回的行都是唯一的
union all则会返回所有SELECT语句中得到所有匹配的行

您可能也喜欢：

MySQL触发器介绍

Mysql存储过程学习笔记--变量、参数、注释

MySQL写入优化

PHP中操作mysql执行存储过程

MySQL视图介绍

无觅

config

; supervisor.
[group:gisapp]
programs=gis-8001,gis-8002,gis-8003,gis-8004

[program:gis-8001]
command=python /home/gis/gis/gisserver.py –port=8001
directory=/home/gis/gis/
autorestart=true
redirect_stderr=true
stdout_logfile=/home/gis/gis/logs/gis_server-8001.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn

[program:gis-8002]
command=python /home/gis/gis/gisserver.py –port=8002
directory=/home/gis/gis/
autorestart=true
redirect_stderr=true
stdout_logfile=/home/gis/gis/gis_server-8002.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
[program:gis-8003]
command=python /home/gis/gis/gisserver.py –port=8003
directory=/home/gis/gis/
autorestart=true
redirect_stderr=true
stdout_logfile=/home/gis/gis/gis_server-8003.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
[program:gis-8004]
command=python /home/gis/gis/gisserver.py –port=8004
directory=/home/gis/gis/
autorestart=true
redirect_stderr=true
stdout_logfile=/home/gis/gis/gis_server-8004.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
怎么让四个端口同时提供服务呢？可以使用web服务神器nginx,nginx自带了负载平衡功能，
可以让这4个服务同时提供服务。

nginx config

upstream gisserver{
server 127.0.0.1:8001;
server 127.0.0.1:8002;
server 127.0.0.1:8003;
server 127.0.0.1:8004;
}
location /tile/ {
proxy_pass http://gisserver;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass_header Set-Cookie;
}
更新：

上述配置可以精简，supervisord配置可以使用变量表示

; supervisor.
[group:gisapp]
programs=gis-web

[program:gis-web]
command=python /home/gis/gis/gisserver.py –port=80%(process_num)02d
directory=/home/gis/gis/
autorestart=true
redirect_stderr=true
stdout_logfile=/home/gis/gis/logs/gis_server-80%(process_num)02d.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
numprocs-4
numprocs_start=1
对不同的服务器，之需要调整numprocs值就可以。

您可能也喜欢：
Python PIL 生成带阴影的缩略图	Httpsqs Python Client	Python Queue 的多线程(multi thread)死锁问题	超级简单Python Socket Server一例	python满足你需要的50个模块
无觅

上面是网上看的，我发现慢查询日志内有很多update和insert的查询，就把innodb_flush_log_at_trx_commit改成了2，效果很明显，改成0会更明显，但安全性比较差。做下面的操作启动mysqld就生效：
vim /etc/my.cnf
innodb_flush_log_at_trx_commit=2

也可以在mysqld运行时执行：
set GLOBAL innodb_flush_log_at_trx_commit = 2

下面是mysql手册上innodb_flush_log_at_trx_commit的解释：
如果innodb_flush_log_at_trx_commit设置为0，log buffer将每秒一次地写入log file中，并且log file的flush(刷到磁盘)操作同时进行；但是，这种模式下，在事务提交的时候，不会有任何动作。如果 innodb_flush_log_at_trx_commit设置为1(默认值)，log buffer每次事务提交都会写入log file，并且，flush刷到磁盘中去。如果innodb_flush_log_at_trx_commit设置为2，log buffer在每次事务提交的时候都会写入log file，但是，flush(刷到磁盘)操作并不会同时进行。这种模式下，MySQL会每秒一次地去做flush(刷到磁盘)操作。注意：由于进程调度策 略问题，这个“每秒一次的flush(刷到磁盘)操作”并不是保证100%的“每秒”。
默认值1是为了ACID (atomicity, consistency, isolation, durability)原子性，一致性，隔离性和持久化的考虑。如果你不把innodb_flush_log_at_trx_commit设置为1，你将获得更好的性能，但是，你在系统崩溃的情况，可能会丢失最多一秒钟的事务数据。当你把innodb_flush_log_at_trx_commit设置 为0，mysqld进程的崩溃会导致上一秒钟所有事务数据的丢失。如果你把innodb_flush_log_at_trx_commit设置为2，只有在操作系统崩溃或者系统掉电的情况下，上一秒钟所有事务数据才可能丢失。InnoDB的crash recovery崩溃恢复机制并不受这个值的影响，不管这个值设置为多少，crash recovery崩溃恢复机制都会工作。

另外innodb_flush_method参数也值得关注，对写操作有影响：
innodb_flush_method： 设置InnoDB同步IO的方式：
1) Default – 使用fsync（）。
2) O_SYNC 以sync模式打开文件，通常比较慢。
3) O_DIRECT，在Linux上使用Direct IO。可以显著提高速度，特别是在RAID系统上。避免额外的数据复制和double buffering（mysql buffering 和OS buffering）。

您可能也喜欢：

MySQL通信协议

mysql内存表heap使用总结

MySQL视图介绍

MySQL 5.1.56 使用 InnoDB Plugin

mysql root帐号丢失解决办法

无觅

1.子查询优化法
先找出第一条数据，然后大于等于这条数据的id就是要获取的数据
缺点：数据必须是连续的，可以说不能有where条件，where条件会筛选数据，导致数据失去连续性

实验下

Sql代码 

1. mysql> set profiling=1;
2. Query OK, 0 rows affected (0.00 sec)
4. mysql> select count(*) from Member;
5. +———-+
6. | count(*) |
7. +———-+
8. |   169566 |
9. +———-+
10. 1 row in set (0.00 sec)
12. mysql> pager grep !~-
13. PAGER set to ’grep !~-’
15. mysql> select * from Member limit 10, 100;
16. 100 rows in set (0.00 sec)
18. mysql> select * from Member where MemberID >= (select MemberID from Member limit 10,1) limit 100;
19. 100 rows in set (0.00 sec)
21. mysql> select * from Member limit 1000, 100;
22. 100 rows in set (0.01 sec)
24. mysql> select * from Member where MemberID >= (select MemberID from Member limit 1000,1) limit 100;
25. 100 rows in set (0.00 sec)
27. mysql> select * from Member limit 100000, 100;
28. 100 rows in set (0.10 sec)
30. mysql> select * from Member where MemberID >= (select MemberID from Member limit 100000,1) limit 100;
31. 100 rows in set (0.02 sec)
33. mysql> nopager
34. PAGER set to stdout
37. mysql> show profiles\G
38. *************************** 1. row ***************************
39. Query_ID: 1
40. Duration: 0.00003300
41. Query: select count(*) from Member
43. *************************** 2. row ***************************
44. Query_ID: 2
45. Duration: 0.00167000
46. Query: select * from Member limit 10, 100
47. *************************** 3. row ***************************
48. Query_ID: 3
49. Duration: 0.00112400
50. Query: select * from Member where MemberID >= (select MemberID from Member limit 10,1) limit 100
52. *************************** 4. row ***************************
53. Query_ID: 4
54. Duration: 0.00263200
55. Query: select * from Member limit 1000, 100
56. *************************** 5. row ***************************
57. Query_ID: 5
58. Duration: 0.00134000
59. Query: select * from Member where MemberID >= (select MemberID from Member limit 1000,1) limit 100
61. *************************** 6. row ***************************
62. Query_ID: 6
63. Duration: 0.09956700
64. Query: select * from Member limit 100000, 100
65. *************************** 7. row ***************************
66. Query_ID: 7
67. Duration: 0.02447700
68. Query: select * from Member where MemberID >= (select MemberID from Member limit 100000,1) limit 100

从结果中可以得知，当偏移1000以上使用子查询法可以有效的提高性能。

2.倒排表优化法
倒排表法类似建立索引，用一张表来维护页数，然后通过高效的连接得到数据

缺点：只适合数据数固定的情况，数据不能删除，维护页表困难

具体请看，http://blog.chinaunix.net/u/29134/showart_1333566.html

3.反向查找优化法
当偏移超过一半记录数的时候，先用排序，这样偏移就反转了

缺点：order by优化比较麻烦，要增加索引，索引影响数据的修改效率，并且要知道总记录数
，偏移大于数据的一半

做下实验，看看性能如何

总记录数：1,628,775
每页记录数： 40
总页数：1,628,775 / 40 = 40720
中间页数：40720 / 2 = 20360

第21000页
正向查找SQL:

Sql代码 

1. SELECT * FROM `abc` WHERE `BatchID` = 123 LIMIT 839960, 40

时间：1.8696 秒

反向查找sql:

Sql代码 

1. SELECT * FROM `abc` WHERE `BatchID` = 123 ORDER BY InputDate DESC LIMIT 788775, 40

时间：1.8336 秒

第30000页
正向查找SQL:

Sql代码 

1. SELECT * FROM `abc` WHERE `BatchID` = 123 LIMIT 1199960, 40

时间：2.6493 秒

反向查找sql:

Sql代码 

1. SELECT * FROM `abc` WHERE `BatchID` = 123 ORDER BY InputDate DESC LIMIT 428775, 40

时间：1.0035 秒

注意，反向查找的结果是是降序desc的，并且InputDate是记录的插入时间，也可以用主键联合索引，但是不方便。

4.limit限制优化法
把limit偏移量限制低于某个数。。超过这个数等于没数据，我记得alibaba的dba说过他们是这样做的

5.只查索引法
http://willko.iteye.com/blog/670120

总结：limit的优化限制都比较多，所以实际情况用或者不用只能具体情况具体分析了。页数那么后，基本很少人看的。。。

您可能也喜欢：

MySQL通信协议

MySQL触发器介绍

PHP中操作mysql执行存储过程

Mysql存储过程学习笔记--变量、参数、注释

MySQL中索引限制

无觅

本博客的域名fendou.org 之前在国内注册的，后来转移到godaddy去了，今年早些时候传言要白名单了，随后真的不能访问，于是乎又将DNS服务器换到国内的DNSPOD，还好DNSPOD解析不需要备案，但是懒得改IP，直接cname到一个还在godaddy上解析的域名，结果这个域名现在也被白名单了，这几天各种纠结的事情缠身，也没怎么看博客，结果就杯具了好几天，现在改成直接使用A记录指向国内的主机，先凑合着用，等周末的时候弄到GAE（Google App Engine）上去吧，由于众所周知的原因，GAE这么好的平台，居然和谐了，还得再弄个境外的服务器做反向代理。另外一个纠结的问题是，本博客主域直接使用的裸域，没有www，GAE不能绑定裸域又是杯具……搬到GAE上最多就是国内不能正常访问而已，好歹数据还在，只要Google 没倒闭，一切都是安全的，Google倒闭的那一天，或许和谐从此消失，换成了民主

您可能也喜欢：
博客大巴（Blogbus）新版正式上线，推出友邻和活动功能	联想IdeaPad发布	ARJ21首飞,真的是中国自主知识产权？	David Archuleta ---Crush	一组很有意思的照片
无觅