IBMPlatformLSF家族安装和配置使用简介.V1.0

《IBMPlatformLSF家族安装和配置使用简介.V1.0》由会员分享，可在线阅读，更多相关《IBMPlatformLSF家族安装和配置使用简介.V1.0（49页珍藏版）》请在装配图网上搜索。

1、word_IBM Platform LSF家族安装和配置简介V1.0版 马雪洁45 / 49目录1集群结构1单纯LSF环境命令行提交11.2LSF+PAC环境WEB提交11.3LSF+PM环境PM提交32LSF安装和根本配置举例3安装步骤3获得LSF和PAC安装包3解压缩lsfinstall安装脚本文件4修改配置文件4执行安装4配置开机自启动4测试安装4启动/停止LSF进程三种方式5测试提交作业5使能root提交作业5修改配置文件后reconfig6日志和debug6配置文件说明6常用命令7配置公平竞争调度策略7添加轮循调度队列7添加层次公平竞争策略8多队列公平竞争策略8使能配置9配置抢占调度

2、策略9配置全局限制策略10限制用户运行的作业数目10限制节点运行作业数目10限制队列作业的运行限制10设定General limits11使能配置12配置提交控制脚本esub12配置资源管理elim示例12汇报home目录空闲大小12汇报root进程数目12汇报应用程序许可证数目13测试elim脚本14添加资源定义和资源地图14查看资源数目143LSF命令行集成应用示例143.1CFD+集成spooling file143.1.1CFD+安装和许可证14集成许可证管理elim14添加CFD+ job starter15添加CFD APP profile173.1.5CFD+命令行提交脚本实例1

3、7集成方式spooling file18安装和许可证18命令行提交脚本实例18的脚本集成bsub命令183.4Platform MPI作业19作业203.6Intel MPI作业20版本不记账方式20版本blaunch记账方式22版本PAM集成方式274安装PAC295使用PAC进展应用程序集成30界面集成过程315.2CFD+集成后界面和后台脚本36在PAC中监控许可证386安装License Scheduler40根本安装测试40根本配置举例40添加许可证服务器地址40映射许可证feature：40使用许可证资源41配置许可证调度策略417常见问题418使用man page419售后技术支

4、持411 集群结构较大的集群都会设计 单独的登录节点，用户只能ssh到登录节点，不能直接ssh到集群的任何主节点和计算节点。同时配置用户在计算节点之间的ssh互信，为了并行作业的运行。登录节点也安装LSF，配置为LSF 静态Client或者MXJ值为0，也即不运行作业的客户端。集群的WEB节点与办公访问局域网一个网段。如需使用浮动client，主节点网卡需要1.1 单纯LSF环境命令行提交1.2 LSF+PAC环境WEB提交用户通过portal提交作业：1.3 LSF+PM环境PM提交2 LSF安装和根本配置举例2.1 安装前的准备工作NIS ready；NFS/GPFS ready；2.2

5、LSF安装步骤Use root to install.Get NIS and NFS/GPFS ready.2.2.1 获得LSF和PAC安装包lsf8.3_linux2.6-glibc2.3-x86_64.tar.Z lsf8.3_lsfinstall_linux_x86_64.tar.Z pac8.3_standard_linux-x64.tar.Z 许可证文件platform_hpc_std2.2.2 解压缩lsfinstall安装脚本文件Put the package under /root/lsf2.2.3首先添加集群管理员lsfadmin。vi ins LSF_TOP=/opt/l

6、sf (安装目录) LSF_ADMINS=lsfadmin (先创建lsfadmin的用户名) LSF_CLUSTER_NAME=platform (集群名称，任意指定) LSF_MASTER_LIST=s2 s3 (LSF管理节点) LSF_ENTITLEMENT_FILE=/root/lsf/platform_hpc_std_entitlement.dat (安装源许可证的地址) LSF_TARDIR=/root/lsf/ (安装源文件包的地址)2.2.4 执行安装2.2.5 配置开机自启动hostsetuprhostsetup2.2.6 测试安装安装目录下的/conf目录Add sour

7、ce profile.lsf to /etc/profileif no rshLSF_RSH=ssh2.2.7 启动/停止LSF进程三种方式rootS2 conf# lsfstartup/lsfstop或者lsadmin limstatup/limshutdownlsadmin resstartup/resshutdownbadmin hstartup/hshutdown或者lsf_daemons start/stoprootS2 conf# lsidIBM Platform LSF Express 8.3 for IBM Platform HPC, May 10 2012Copyright

8、Platform puting Inc., an IBM pany, 1992-2012.US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.My cluster name is platformMy master name is s2You have new mail in /var/spool/mail/rootrootS2 conf# lsloadHOST_NAME status r15s r

9、1m r15m ut pg ls it tmp swp mems2 ok 0.0 0.0 0.0 1% 0.0 1 0 151G 20G 61Gs4 ok 0.0 0.0 0.0 2% 0.0 1 2 183G 20G 62Gs6 ok 0.0 0.0 0.0 3% 0.0 1 2 3734M 2G 30Gs5 ok 0.0 0.0 0.0 5% 0.0 1 2 3468M 2G 30G2.2.8 测试提交作业bsub sleep 1000002.2.9 使能root提交作业enable root to submit job: LSF_ROOT_REX=local重启LSF进程。2.2.10

10、修改配置文件后reconfig修改lsf.*配置文件后lsadmin reconfig修改lsb.*配置文件后badmin reconfig局部参数需要重启LSF主调度或者其他进程：badmin mbdrestart; lsadmin limrestart; lsadmin resrestart; badmin hrestart2.2.11 日志和debugFind the logs under log directory. LSF will run mainly 3 processes on each node, on master node will have 2 more.Master:

11、 lim,res,sbatchd,mbatchd,mbschedpute:lim,res,sbatchdTurn on debug in mand line:Run lim -2 directly on node to check why lim not startup.2.3 配置文件说明目录/etc/init.d:服务自启动脚本目录/apps/platform/8.3/lsf/conf:lsf.conf lsf配置文件 集群配置文件 共享资源定义文件./lsbatch/cluster83/configdir/lsb.* 调度系统配置文件lsb.users lsf用户与用户组配置文件lsb.

12、queues lsf队列配置文件调度参数配置文件应用配置文件机器与机器组配置文件资源配置文件lsb.modules lsf模块配置文件2.4 常用命令bsub:提交作业；bjobs:查看作业信息；bhist:查看作业历史；lshosts:查看节点静态资源；bhosts，lsload:查看节点状态和资源信息；bqueues ：查看队列配置；blimits：查看限制limit信息；lsid:集群版本和主节点；bmod:修改bsub option；等等。2.5 基于资源的调度策略bsub R “(type=LINUX2.4 & r1m 2.0)|(type=AIX & r1m 1.0)或者在队列ls

13、b.queues或者lsb.application文件定义：RES_REQ=select(type=LINUX2.4 & r1m 2.0)|(type=AIX & r1m =300 & mem500 orderswap:memrusageswap=300,mem=500 job1bsub R rusagemem=500:app_lic_v2=1 | mem=400:app_lic_v1.5=1 job1bsub R selecttype=any & swp=300 & mem500 ordermem job12.6 配置公平竞争调度策略2.6.1 添加轮循调度队列Modify lsb.queu

14、es, add followingBegin QueueQUEUE_NAME = roundRobin PRIORITY = 40FAIRSHARE = USER_SHARESdefault,1#USERS = userGroupA Define your own usergroupEnd QueueRun badmin reconfig to enable the change.Run bqueues l to check the queues configure2.6.2 添加层次公平竞争策略Add following queue to add hierarchicalshare poli

15、cy:Begin QueueQUEUE_NAME = hierarchicalShare PRIORITY = 40USERS = userGroupB userGroupC FAIRSHARE = USER_SHARESuserGroupB,7 userGroupC,3End Queue2.6.3 多队列公平竞争策略在lsb.queues中添加如下队列，注意节点组和用户组定义。Begin QueueQUEUE_NAME = verilog DESCRIPTION = master queue definition cross-queuePRIORITY = 50FAIRSHARE = USE

16、R_SHARESuser1,100 default,1FAIRSHARE_QUEUES = normal shortHOSTS = hostGroupC# resource contention #RES_REQ = rusageverilog = 1End QueueBegin QueueQUEUE_NAME = shortDESCRIPTION = short jobs PRIORITY = 70 # highestHOSTS = hostGroupC RUNLIMIT = 5 10End QueueBegin QueueQUEUE_NAME = normalDESCRIPTION = d

17、efault queuePRIORITY = 40 # lowestHOSTS = hostGroupC End Queue2.6.4 使能配置badmin reconfig提交作业，并查看队列的用户动态优先级变化：bqueues rl normal2.7 配置抢占调度策略配置最根本的slots抢占：Begin QueueQUEUE_NAME = shortPRIORITY = 70HOSTS = hostGroupC # potential conflictPREEMPTION = PREEMPTIVEnormalEnd QueueBegin QueueQUEUE_NAME = normal

18、PRIORITY = 40HOSTS = hostGroupC # potential conflictPREEMPTION = PREEMPTABLEshortEnd Queue向两个队列提交作业，查看被preempt的作业的pending原因。2.8 配置全局限制策略2.8.1 限制用户运行的作业数目在lsb.users文件中添加：Begin User USER_NAME MAX_JOBS JL/P user1 4 - user2 2 1 user3 - 2 groupA 8 - groupB 1 1 Default 2 - End User2.8.2 限制节点运行作业数目在lsb.hos

19、ts文件中：Begin HostHOST_NAME MXJ JL/Uhost1 4 2host2 2 1host3 ! -End Host2.8.3 限制队列作业的运行限制在lsb.queues中添加：Begin QueueQUEUE_NAME = myQueue HJOB_LIMIT = 2PJOB_LIMIT = 1UJOB_LIMIT = 4HOSTS = hostGroupA USERS = userGroupA End Queue2.8.4 设定General limits在lsb.resources文件定义全局general limits示例：Begin LimitUSERSQUE

20、UESHOSTS SLOTS MEM SWP user1 -hostB - - 20% user2normalhostA - 20 - End LimitBegin Limit NAME = limit1 USERS = user1 PER_HOST = hostA hostC TMP = 30% SWP = 50% MEM = 10%End LimitBegin Limit PER_USER QUEUES HOSTS SLOTS MEM SWP TMP JOBS groupA - hgroup1 - - - - 2 user2 normal - - 200 - - - - short - -

21、 - - - 200 End Limit 2.8.5 使能配置badmin reconfig2.9 配置提交控制脚本esub全局esub脚本在作业被提交是调用，可以被自动的或者显式的调用从而控制用户作业提交的行为。编辑esub.project文件在$LSF_SERVERDIR下面chmod为可执行：#!/bin/shif _$LSB_SUB_PARM_FILE != _ ; then . $LSB_SUB_PARM_FILE if _$LSB_SUB_PROJECT_NAME = _ ; then echo You must specify a project! &2 exit $LSB_SU

22、B_ABORT_VALUE fifiexit 0在lsf.conf中定义LSB_ESUB_METHOD=project2.10 配置资源管理elim示例2.10.1 汇报home目录空闲大小编辑elim文件elim.home，放置在$LSF_SERVERDIR下面。chmod为可执行。#!/bin/shwhile true ; do home=df -k /home | tail -1 | awk printf %4.1f, $4/(1024*1024) echo 1 home $home sleep 30done2.10.2 汇报root进程数目编辑elim.root，放置在$LSF_SER

23、VERDIR下面。chmod为可执行。#!/bin/shwhile true ; do root=ps -ef | grep -v grep | grep -c root echo 1 rootprocs $root sleep 30done2.10.3 汇报应用程序许可证数目#!/bin/shlic_X=0 ; num=0while true ; do # only want the master to gather lic_X if $LSF_MASTER = Y ; then lic_X=lmstat a c lic_X.dat | grep .&2 fi # only want tra

24、ining8, training1 to gather simpton licenses if hostname = training8 o hostname = training1 ; then num=lmstat a c simpton_lic.dat | grep .&2 fi # all hosts including master, will gather the following root=ps efw | grep v grep | grep c root1&2 tmp=df k /var/tmp | grep var | awk print $4 /1024&2if $LS

25、F_MASTER = Y ; then echo 4 lic_X $lic_X simpton $num rtprc $root tmp $tmp else echo 3 simpton $num rtprc $root tmp $tmp fi sleep 60done2.10.4 测试elim脚本直接运行./elim.root查看elim输出是否正确。2.10.5 添加资源定义和资源地图在lsf.shared文件中添加rootprocs 定义，并在lsf.cluster resources Map中添加资源和节点的映射关系。使能配置：lsadmin reconfig; badmin reco

26、nfig2.10.6 查看资源数目lsload l3 LSF命令行集成应用示例本节例举几个应用的不同集成方式。使用spooling文件或者bsub命令行都可以自由转换。3.1 CFD+集成spooling file3.1.1 CFD+安装和许可证安装路径：ln-3620-4许可证：许可证服务器：ln-3620-4启动许可证服务器：hpcadminmn-3650 jessi$ ssh ln-3620-4确认许可证服务器是否正常运行：3.1.2 集成许可证管理elim添加elim方法：elim全集群只需运行一个，因此只在头节点放置elim脚本即可在头节点：cd $LSF_SERVERDIR添加如下

27、文件：elim.lic:rootmn-3650 jessi# cd $LSF_SERVERDIRrootmn-3650 etc# pwd#!/bin/shtotallicences=/gpfs/software/cfdpp/mbin/lmutil lmstat -a -c /gpfs/software/cfdpp/mbin/Metap.lic |grep Users of CFD+_SOLV_Ser | /bin/cut -d -f7while truedousedlicences=/gpfs/software/cfdpp/mbin/lmutil lmstat -a -c /gpfs/soft

28、ware/cfdpp/mbin/Metap.lic | /bin/grep Users of CFD+_SOLV_Ser | /bin/cut -d -f13 cfd_lic=$( $totallicences-$usedlicences )echo 1 cfd_lic $cfd_lic/bin/sleep 30done修改如下的配置文件：rootmn-3650 etc# vi $LSF_ENVDIR/添加如下一行：cfd_lic Numeric 30 Y (CFD+ License)rootmn-3650 etc# vi $LSF_ENVDIR/lsf.cluster在resourcemap

29、一段添加如下一行：Begin ResourceMap RESOURCENAME LOCATION cfd_lic allhostid default rootmn-3650 etc# lsadmin reconfig; badmin reconfig3.1.3 添加CFD+ job starter如果使用spooling file可不用添加。Portal集成方式使用添加jobstarter可执行文件：hpcadminmn-3650 jessi$ cat /opt/lsf/jobstarter/cfd_starter #!/bin/shMPI_RUN=/gpfs/software/cfdpp/h

30、pmpi/bin/mpiruncase $PRESSION in SINGLE_PRESSION) ; DOUBLE_PRESSION) ; esacCMD=$* -hostfile $LSB_DJOB_HOSTFILE $CFD_CMDeval $CMD3.1.4 添加CFD APP profilerootmn-3650 etc# vi $LSF_ENVDIR/添加如下配置：Begin ApplicationNAME = cfdJOB_STARTER=/opt/lsf/jobstarter/cfd_starterRES_REQ=rusagecfd_lic=1End Applicationba

31、dmnin reconfig使得此文件生效，使用bapp l cfd查看是否成功：rootmn-3650 bin# bapp -l cfdAPPLICATION NAME: cfd - No description provided.STATISTICS: NJOBS PEND RUN SSUSP USUSP RSV 12 12 0 0 0 0PARAMETERS:JOB_STARTER: /opt/lsf/jobstarter/cfd_starterRES_REQ: rusagecfd_lic=13.1.5 CFD+命令行提交脚本实例#!/bin/sh#BSUB -n 12#BSUB -ap

32、p cfd#BSUB -R rusagecfd_lic=1cd /gpfs/software/cfd+/test/ogive/然后bsub cfd.sh 提交作业。3.2 GAUSSIAN集成方式spooling file3.2.1 Gaussian安装和许可证路径：/gpfs/software/Gaussian/许可证：无许可证版本，单个作业只能单机运行。3.2.2 Gaussian命令行提交脚本实例#!/bin/sh#BSUB -q qchem#BSUB -n 4#BSUB -R spanhosts=1#BSUB -cwd .JOB=Full_codes_112_ipr_C1_0588.J

33、OBNAME=basename $JOB .export g03root=/gpfs/software/Gaussianexport GAUSS_SCRDIR=/tmp/gpfs/software/Gaussian/g03/g03 $JOBNAME.log提交作业：3.3 Abaqus的脚本集成bsub命令#!/bin/sh#export ABAQUS_CMD=/gpfs/software/Abaqus/mands/abaqusexport LM_LICENSE_FILE=/gpfs/software/Abaqus/License/abq612.lic#指定cpu number,注意要与bsu

34、b命令行中-n指定的cpu个数一致export NCPU=16#指定输入文件#指定作业名export JOB_NAME=abaqus_job3$ABAQUS_CMD job=$JOB_NAME cpus=$NCPU input=$INP_INPUT_FILE2)通过LSF提交#进入输入数据所在目录，执行bsub命令3.4 Amber作业blaunch集成，可记账针对intelmpi，编写mpdboot.lsf脚本。变为可执行，放置在$LSF_SERVERDIR下面编写提交作业脚本：ymeimnis test$ cat new.sh #!/bin/sh#BSUB -q small#BSUB -n

35、 128#BSUB -J IMPI#BSUB -x #export PATH=/gpfs01/software/intel/impi/4.1.0.024/intel64/bin:$PATHexport I_MPI_DEVICE=ssm#export I_MPI_FABRICS=shm:ofa#export I_MPI_FAST_STARTUP=1#export I_MPI_DEVICE=rdssmmpdallexit提交作业：3.5 Platform MPI作业3.5.1 安装Platform MPI确认用户无密码访问ssh OK。安装Platform MPI到共享目录下：sh platfor

36、m_mpi-08.3.0.0-0320r.x64.sh -installdir=/opt/pmpi norpm如果缺失C piler，执行：yum install gcc3.5.2 LSF外面验证安装OK设置环境变量：export MPI_REMSH=ssh -xexport MPI_ROOT=/opt/pmpi/opt/ibm/platform_mpi/编译helloworld示例程序：rootserver3 help# /opt/pmpi/opt/ibm/platform_mpi/bin/mpirun -f ./help/hosts warning: MPI_ROOT /opt/pmpi/

37、opt/ibm/platform_mpi/ != mpirun path /opt/pmpi/opt/ibm/platform_mpiHello world! Im 1 of 4 on server3Hello world! Im 0 of 4 on server3Hello world! Im 3 of 4 on puter007Hello world! Im 2 of 4 on puter007rootserver3 help# cat ./help/hosts -h server3 -np 2 /opt/pmpi/opt/ibm/platform_mpi/help/helloworld

38、-h puter007 -np 2 /opt/pmpi/opt/ibm/platform_mpi/help/helloworld3.5.3 通过LSF提交export MPI_REMSH=blaunch$ mpirun -np 4 -IBV /helloworld$ mpirun -np 32 -IBV /helloworld$ mpirun -np 4 -TCP /helloworld或者 rootserver3 conf# bsub -o %J.out -e.%J.err -n 4 /opt/pmpi/opt/ibm/platform_mpi/bin/mpirun -lsb_mcpu_ho

39、sts /opt/pmpi/opt/ibm/platform_mpi/help/helloworldJob is submitted to default queue .rootserver3 conf# bjobsJOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME210 root PEND normal server3 *elloworld May 9 10:55 rootserver3 conf# cat 210.out Sender: LSF System Subject: Job 210: in cluster

40、DoneJob was submitted from host by user in cluster .Job was executed on host(s) , in queue , as user in cluster . was used as the home directory. was used as the working directory.Started at Thu May 9 18:49:06 2013Results reported at Thu May 9 18:49:07 2013Your job looked like:-# LSBATCH: User input

41、/opt/pmpi/opt/ibm/platform_mpi/bin/mpirun -lsb_mcpu_hosts /opt/pmpi/opt/ibm/platform_mpi/help/helloworld-Successfully pleted.Resource usage summary: CPU time : 0.23 sec. Max Memory : 2 MB Average Memory : 2.00 MB Total Requested Memory : - Delta Memory : - (Delta: the difference between total reques

42、ted memory and actual max usage.) Max Swap : 36 MB Max Processes : 1 Max Threads : 1The output (if any) follows:Hello world! Im 2 of 4 on puter007Hello world! Im 0 of 4 on puter007Hello world! Im 1 of 4 on puter007Hello world! Im 3 of 4 on puter007PS:Read file for stderr output of this job.或者更多参数$ /

43、opt/platform_mpi/bin/mpirun -np 120 -ibv -hostlist -22-001-22-002-22-003-22-004-22-005-22-006-22-007-22-008-22-009-22-010 /data/hello_world如果希望 MPI作业不通过LSF提交运行，修改MPI_USELF环境变量为n3.6 Openmpi作业下载openmpi软件包./configure LIBS=-ldl -with-lsf=yes -prefix=/usr/local/ompi/Open mpi1.3.2 之上版本已经于LSF blaunch严密集成。提

44、交openmpi作业：bsub n2 o %J.out 3.7 Intel MPI作业3.7.1 Express版本不记账方式如果需要对作业记账，需要使用blaunch的集成方式。export PATH=/gpfs/software/intel/poserxe/bin/:/gpfs/software/intel/mpi_41_0_024/include:/gpfs/software/intel/mpi_41_0_024/bin64:/gpfs/software/intel/poserxe/mkl:$PATHsource /gpfs/software/intel/poserxe/bin/pile

45、rvars.sh intel64source /gpfs/software/intel/poserxe/mkl/bin/mklvars.sh intel64MPI测试程序 #include mpi.h #include #include int main(int argc,char*argv) int myid, numprocs; int namelen; char processor_nameMPI_MAX_PROCESSOR_NAME; MPI_Init(&argc,&argv); MPI_m_rank(MPI_M_WORLD,&myid); MPI_m_size(MPI_M_WORLD

46、,&numprocs); MPI_Get_processor_name(processor_name,&namelen); fprintf(stderr,Hello World! Process %d of %d on %sn,myid, numprocs, processor_name); MPI_Finalize(); 命令执行，TCP协议命令执行，IB网络命令执行，Debug模式LSF提交脚本bsub_intelmpi_ib.sh #!/bin/sh#BSUB -cwd .#BSUB -R spanptile=4提交作业：bsub bsub_intelmpi_ib.sh 3.7.2 Ex

47、press版本blaunch记账方式3.7.2.1#!/usr/bin/env pythonmpdboot for LSF -f | -hostfile hostfile -i | -ifhn=alternate_interface_hostname_of_ip_address -f | -hostfile hostfile -himport reimport stringimport time import sysimport getoptfrom time import ctimefrom os import environ, pathfrom sys import argv, exit,

48、 stdoutfrom popen2 import Popen4from socket import gethostname, gethostbynamedef mpdboot(): # change me MPI_ROOTDIR = /opt/intel/impi/4.0.0.025 # mpdCmd = %s/bin/mpd % MPI_ROOTDIR mpdtraceCmd = %s/bin/mpdtrace % MPI_ROOTDIR mpdtraceCmd2 = %s/bin/mpdtrace -l % MPI_ROOTDIR nHosts = 1 host = ip = local

49、Host = localIp = found = False MAX_WAIT = 5 t1 = 0 hostList = hostTab = cols = hostArr = hostfile = environ.get(LSB_DJOB_HOSTFILE) binDir = environ.get(LSF_BINDIR) if environ.get(LSB_MCPU_HOSTS) = None or hostfile = None or binDir = None: print not running in LSF exit (-1) rshCmd = binDir + /blaunch

50、 p = re.pile(S+_d+s+(d+.d+.d+.d+)# try: opts, args = getopt.getopt(sys.argv1:, hf:i:, help, hostfile=, ifhn=) except getopt.GetoptError, err: print str(err) usage() sys.exit(-1) fileName = None ifhn = None for o, a in opts: if o = -v: version(); sys.exit() elif o in (-h, -help): usage() sys.exit() e

51、lif o in (-f, -hostfile): fileName = a elif o in (-i, -ifhn): ifhn = a else: print option %s unrecognized % o usage() sys.exit(-1) if fileName = None: if ifhn != None: print -ifhn requires a host file containing hostname ifhn=alternate_interface_hostname_of_ip_addressn sys.exit(-1) # use LSB_DJOB_HO

52、STFILE fileName = hostfile localHost = gethostname() localIp = gethostbyname(localHost) pifhn = re.pile(w+s+ifhn=d+.d+.d+.d+) #pifhn = re.pile(S+ifhn=d+.d+.d+.d+) try: # check the hostfile machinefile = open(fileName, r) for line in machinefile: if not line or line0 = #: continue line = re.split(#, line)0 line = line.strip() if not line: continue if not pifhn.match (lin