文号
t79253
阅读
9790
回复
15
收藏
6
【数据分析】网页数据提取
2016/03/14
IP:北京
最近需要用到XXXXXXXXXXXXXXXXXXXXXXX里的左侧菜单栏的监测曲线里的PM2.5数据,
1.jpg
之前都是逐个手工输入excel,遇到数据量大的时候就很费时费力了,而且页面会定时刷新,刷新后需要重新设定检索时间段,非常麻烦,所以希望能从网页中提取出不同时间点下的PM2.5数据。

先试了excel的数据导入,但是显示不了窗口,
2.jpg


但是可以输出SVG文件,右键可以查看源代码,用SVGdeveloper确定了坐标轴和数据点对应的代码,但是理解不了,烦请各位大侠给讲讲,下面是2016年3月13日00:00时单点SVG对应的代码
3.jpg


[blockquote]<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"><svg xmlns:xlink="XXXXXXXXXXXXXXXXXXXXXXXXXXXX" version="1.1" xmlns="XXXXXXXXXXXXXXXXXXXXXXXXXX" width="600" height="400">
        <desc>Created with Highcharts 3.0.9</desc>
        <defs>
                <clipPath id="highcharts-52">
                        <rect fill="none" x="0" y="0" width="511" height="309"></rect>
                </clipPath>
        </defs>
        <rect rx="0" ry="0" fill="#FFFFFF" x="0" y="0" width="600" height="400"></rect>
        <g class="highcharts-grid"></g>
        <g class="highcharts-grid">
                <path fill="none" d="M 69 57.5 L 580 57.5" stroke="#C0C0C0" stroke-width="1" opacity="1"></path>
                <path fill="none" d="M 69 135.5 L 580 135.5" stroke="#C0C0C0" stroke-width="1" opacity="1"></path>
                <path fill="none" d="M 69 213.5 L 580 213.5" stroke="#C0C0C0" stroke-width="1" opacity="1"></path>
                <path fill="none" d="M 69 290.5 L 580 290.5" stroke="#C0C0C0" stroke-width="1" opacity="1"></path>
                <path fill="none" d="M 69 367.5 L 580 367.5" stroke="#C0C0C0" stroke-width="1" opacity="1"></path>
        </g>
        <g class="highcharts-axis">
                <path fill="none" d="M 324.5 367 L 324.5 372" stroke="#C0D0E0" stroke-width="1" opacity="1"></path>
                <path fill="none" d="M 69 367.5 L 580 367.5" stroke="#C0D0E0" stroke-width="1" visibility="visible"></path>
        </g>
        <g class="highcharts-axis">
                <text x="28.25" y="212.5" style="font-family:'lucida grande', 'lucida sans unicode', verdana, arial, helvetica, sans-serif;font-size:12px;color:#4d759e;font-weight:bold;fill:#4d759e;" text-anchor="middle" transform="translate(0,0) rotate(270 28.25 212.5)" visibility="visible">
                        <tspan x="28.25">PM2.5(ug/m3)</tspan>
                </text>
        </g>
        <g class="highcharts-series-group">
                <g class="highcharts-series" visibility="visible" transform="translate(69,58) scale(1 1)" clip-path="url(#highcharts-52)">
                        <path fill="none" d="M 245.5 71.07000000000002 L 265.5 71.07000000000002" stroke-linejoin="round" visibility="visible" stroke-opacity="0.0001" stroke="rgb(192,192,192)" stroke-width="22" class=" highcharts-tracker"></path>
                </g>
                <g class="highcharts-markers highcharts-tracker" visibility="visible" transform="translate(69,58) scale(1 1)" clip-path="none">
                        <path fill="#2f7ed8" d="M 255 67.07000000000002 C 260.328 67.07000000000002 260.328 75.07000000000002 255 75.07000000000002 C 249.672 75.07000000000002 249.672 67.07000000000002 255 67.07000000000002 Z"></path>
                </g>
        </g>
        <text x="295" y="25" style="font-family:'lucida grande', 'lucida sans unicode', verdana, arial, helvetica, sans-serif;font-size:16px;color:#274b6d;fill:#274b6d;" text-anchor="middle" class="highcharts-title">
                <tspan x="295">北京PM2.5小时变化趋势</tspan>
        </text>
        <text x="295" y="40" style="font-family:'lucida grande', 'lucida sans unicode', verdana, arial, helvetica, sans-serif;font-size:12px;color:#4d759e;fill:#4d759e;" text-anchor="middle" class="highcharts-subtitle">
                <tspan x="295">最小值:154  平均值:154.0  最大值:154</tspan>
        </text>
        <g class="highcharts-axis-labels">
                <text x="324.5" y="381" style="font-family:'lucida grande', 'lucida sans unicode', verdana, arial, helvetica, sans-serif;font-size:11px;color:#666;cursor:default;fill:#666;" text-anchor="middle" opacity="1">
                        <tspan x="324.5">03-13</tspan>
                </text>
        </g>
        <g class="highcharts-axis-labels">
                <text x="61" y="370" style="font-family:'lucida grande', 'lucida sans unicode', verdana, arial, helvetica, sans-serif;font-size:11px;color:#666;cursor:default;fill:#666;" text-anchor="end" opacity="1">
                        <tspan x="61">0</tspan>
                </text>
                <text x="61" y="292.75" style="font-family:'lucida grande', 'lucida sans unicode', verdana, arial, helvetica, sans-serif;font-size:11px;color:#666;cursor:default;fill:#666;" text-anchor="end" opacity="1">
                        <tspan x="61">50</tspan>
                </text>
                <text x="61" y="215.5" style="font-family:'lucida grande', 'lucida sans unicode', verdana, arial, helvetica, sans-serif;font-size:11px;color:#666;cursor:default;fill:#666;" text-anchor="end" opacity="1">
                        <tspan x="61">100</tspan>
                </text>
                <text x="61" y="138.25" style="font-family:'lucida grande', 'lucida sans unicode', verdana, arial, helvetica, sans-serif;font-size:11px;color:#666;cursor:default;fill:#666;" text-anchor="end" opacity="1">
                        <tspan x="61">150</tspan>
                </text>
                <text x="61" y="61" style="font-family:'lucida grande', 'lucida sans unicode', verdana, arial, helvetica, sans-serif;font-size:11px;color:#666;cursor:default;fill:#666;" text-anchor="end" opacity="1">
                        <tspan x="61">200</tspan>
                </text>
        </g>
        <g class="highcharts-tooltip" style="cursor:default;padding:0;white-space:nowrap;" transform="translate(0,-999)">
                <rect rx="3" ry="3" fill="none" x="0.5" y="0.5" width="16" height="16" fill-opacity="0.85" stroke="black" stroke-opacity="0.049999999999999996" stroke-width="5" transform="translate(1, 1)"></rect>
                <rect rx="3" ry="3" fill="none" x="0.5" y="0.5" width="16" height="16" fill-opacity="0.85" stroke="black" stroke-opacity="0.09999999999999999" stroke-width="3" transform="translate(1, 1)"></rect>
                <rect rx="3" ry="3" fill="none" x="0.5" y="0.5" width="16" height="16" fill-opacity="0.85" stroke="black" stroke-opacity="0.15" stroke-width="1" transform="translate(1, 1)"></rect>
                <rect rx="3" ry="3" fill="rgb(255,255,255)" x="0.5" y="0.5" width="16" height="16" fill-opacity="0.85"></rect>
                <text x="8" y="21" style="font-family:'lucida grande', 'lucida sans unicode', verdana, arial, helvetica, sans-serif;font-size:12px;color:#333333;fill:#333333;"></text>
        </g>
        <text x="519" y="43" xml:space="preserve" font-family="Microsoft YaHei UI" font-size="12" fill="#FFFFFF" stroke="none" stroke-opacity="0" ></text>
</svg>
[/blockquote]下面这些是坐标轴的
[blockquote]<g class="highcharts-grid" ><path fill="none" d="M 69 57.5 L 580 57.5" stroke="#C0C0C0" stroke-width="1"  opacity="1">
</path><path fill="none" d="M 69 135.5 L 580 135.5" stroke="#C0C0C0" stroke-width="1"  opacity="1">
    
</path><path fill="none" d="M 69 213.5 L 580 213.5" stroke="#C0C0C0" stroke-width="1"  opacity="1">
    
</path><path fill="none" d="M 69 290.5 L 580 290.5" stroke="#C0C0C0" stroke-width="1"  opacity="1">
    
</path><path fill="none" d="M 69 367.5 L 580 367.5" stroke="#C0C0C0" stroke-width="1"  opacity="1"></path></g>[/blockquote]这个是坐标点的代码:
[blockquote]<path fill="#2f7ed8" d="M 255 67.07000000000002 C 260.328 67.07000000000002 260.328 75.07000000000002 255 75.07000000000002 C 249.672 75.07000000000002 249.672 67.07000000000002 255 67.07000000000002 Z">[/blockquote]分析到这里时间格式和PM2.5浓度搞不出来,恳请各位大侠相助,谢谢 4.jpg
huojunan2作者
10年9个月前IP:北京
812454
4楼
引用 radio:
可以从 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 这里获得数据
用post方法向这个地址传送以下参数:
city:"5p2t5bee"             //"杭州"的ba...
感谢,真是帮大忙了,这是取得了数据接口么
本人完全的小白
引用
评论
huojunan2作者
10年9个月前IP:北京
812514
10楼
引用 radio:
# -*- coding:utf8 -*-
import urllib2,urllib,random,string,base64,time,datetime,re
import string
timestart = datetime....
这个python脚本我应该在那种环境下运行?python里,还是其他地方?
引用
评论
huojunan2作者
10年9个月前IP:北京
812615
13楼
引用 radio:
不好意思没看清主贴。。楼主要获取北京的数据得把第10行的'5p2t5bee'改成‘北京’的编码'5YyX5Lqs'才行蛤
29行的'杭州'也换成'北京'
谢谢
引用
评论
huojunan2作者
10年9个月前IP:北京
812827
14楼
引用 radio:
可以从 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 这里获得数据
用post方法向这个地址传送以下参数:
city:"5p2t5bee"             //"杭州"的ba...
post方法能讲解下么?
引用
评论
笔记
{{note.content}}
{{n.user.username}}
{{fromNow(n.toc)}} {{n.status === noteStatus.disabled ? "已屏蔽" : ""}} {{n.status === noteStatus.unknown ? "正在审核" : ""}} {{n.status === noteStatus.deleted ? '已删除' : ''}}
  • 编辑
  • 删除
  • {{n.status === 'disabled' ? "解除屏蔽" : "屏蔽" }}
我也是有底线的