clair镜像扫描的实现
一、前言
clair扫描的相关基础请先移步我的另外一篇文章镜像安全扫描工具clair与clairctl
这次我们采用clair api方式的扫描，基本思路是
- 打包镜像
- 解压tar包至tomcat的ROOT目录，得到每一个镜像的分层文件及描述文件
- 调用每个镜像的分层描述文件
- 使用clair api按照描述文件中提供的层序提交分层至clair中
- clair依照漏洞-特征数据库进行比对
- 使用clair api并携带最后一层的id，查询漏洞数据
- 删除解压目录
二、启动容器
我们需要三个容器，分别是
- Tomcat：给镜像的分层提供下载地址
docker run -d --name tomcat8 -p 6062:8080 -v /opt/deploy/clair/scanImage:/usr/local/tomcat/webapps/ROOT tomcat:jdk8-corretto - Postgres：存放漏洞-特征匹配数据
docker run -d -e POSTGRES_PASSWORD="password" -p 5432:5432 -v /opt/deploy/clair/postgres/data:/var/lib/postgresql/data --name postgres vmware/postgresql-photon:v1.5.0 - Clair：扫描工具
新建目录/opt/deploy/clair/config,在该目录下新建文件config.yml
database:
type: pgsql
options:
source: postgresql://postgres:password@127.0.0.1:5432/postgres?sslmode=disable
cachesize: 16384
api:
port: 6060
healthport: 6061
timeout: 900s
updater:
interval: 2h
notifier:
attempts: 3
renotifyinterval: 2h
docker run --net=host -d -p 6060-6061:6060-6061 -v /opt/deploy/clair/config:/config quay.io/coreos/clair:v2.0.1 -config=/config/config.yml 其中6060为扫描端口，6061为健康检查端口
注意，Postgres必须在clair之前启动，否则clair会连不上数据库，直接停止容器。
clair与postgres正常启动后，clair会自动去拉取漏洞数据插入数据库中，这一过程往往持续几个小时，且由于网络原因，大量漏洞数据无法拉取，此时需要另辟蹊径。
三、全部步骤
（1）检查工作
- 判断该镜像所在的仓库是否正常，否则无法拉取该镜像
- 判断tomcat、clair、postgres容器是否正常运行中
（2）调用脚本
private String executeRemoteShell() {
String result = null;
SSH ssh = new SSH(主机ip, 用户名, 密码);
try {
boolean isConnect = ssh.connect();
if (isConnect) {
result = ssh.executeWithResult(脚本全路径，传入参数);
}
} catch (IOException e) {
e.printStackTrace();
}
return result;
}
#!/bin/bash
baseDir=/opt/deploy/clair
if [ $? -eq 0 ];then
docker pull ${1} > /dev/null 2>&1
imageName=`echo ${1} | sed 's!\/!_!g'` > /dev/null 2>&1
imageDir="${imageName}" > /dev/null 2>&1
mkdir ${baseDir}/scanImage/${imageDir} > /dev/null 2>&1
docker save -o ${baseDir}/scanImage/${imageDir}/${imageDir}.tar ${1} > /dev/null 2>&1
tar -xvf ${baseDir}/scanImage/${imageDir}/${imageDir}.tar -C ${baseDir}/scanImage/${imageDir}/ > /dev/null 2>&1
cat ${baseDir}/scanImage/${imageDir}/manifest.json
exit 0
else
echo 'scan image failed!'
exit 6
fi （3）使用clair api提交镜像分层数据，postLayers为false时，此时已经提交过分层数据，可以利用最后一层的id直接查询漏洞。（postgres会将已经查询过的镜像漏洞存储，提交完分层数据后，可以使用最后一层的id获取漏洞数据）
//将镜像的分层文件提交至clair，进行分析并返回结果
private ImageScanResult getScanResultFromClair(ImageScan imageScan, boolean postLayers) throws Exception {
String imageName = imageScan.getImageName();
String imageTag = imageScan.getImageTag();
//1.获取manifest层序描述文件内容
String result = RestTemplateUtils.get(getManifestPath(imageName, imageTag));
JSONArray jsonArray = JSONArray.parseArray(result);
if (jsonArray == null || jsonArray.size() == 0) {
throw new Exception("无法获取或解析镜像层序的描述文件");
}
JSONObject jsonObject = (JSONObject) jsonArray.get(0);
@SuppressWarnings("unchecked")
List<String> layers = (List<String>) jsonObject.get("Layers");
//提交的api
String clairApi = "http://" + systemConfig.getRelayServerIp() + ":" + systemConfig.getClairPort() + "/v1/layers";
if (postLayers) {
//上一层id
String beforePath = "";
//2.向clair逐层提交
for (String layer : layers) {
String name = layer.substring(0, layer.indexOf("/"));
String path = getAbsoluteLayerPath(layer, imageScan.getImageName(), imageScan.getImageTag());
String parentName = beforePath;
String format = "Docker";
beforePath = name;
Map<String, Map<String, String>> postParameter = new HashMap<>();
Map<String, String> layerParameter = new HashMap<>();
layerParameter.put("Name", name);
layerParameter.put("Path", path);
layerParameter.put("ParentName", parentName);
layerParameter.put("Format", format);
postParameter.put("Layer", layerParameter);
JSONObject layerJSONObject = JSONObject.parseObject(JSONObject.toJSONString(postParameter));
RestTemplateUtils.post(clairApi, layerJSONObject, null);
}
}
//3.所有层提交完毕后，使用最后一层进行查询漏洞
int last = layers.size() - 1;
String lastName = layers.get(last).substring(0, layers.get(last).indexOf("/"));
String leak = RestTemplateUtils.get(clairApi + "/" + lastName + "?features&vulnerabilities");
JSONObject leakJSONObject = JSONObject.parseObject(leak);
JSONObject leakLayerJSONObject = leakJSONObject.getJSONObject("Layer");
JSONArray featuresJSONArray = leakLayerJSONObject.getJSONArray("Features");
List<Feature> featureList = new ArrayList<>();
Feature feature;
for (Object f : featuresJSONArray) {
JSONObject featureJSONObject = (JSONObject) f;
feature = new Feature();
String name = featureJSONObject.getString("Name");
String namespaceName = featureJSONObject.getString("NamespaceName");
String versionFormat = featureJSONObject.getString("VersionFormat");
String version = featureJSONObject.getString("Version");
String addedBy = featureJSONObject.getString("AddedBy");
List<Vulnerability> vulnerabilities = new ArrayList<>();
JSONArray vulnerabilitiesJSONArray = featureJSONObject.getJSONArray("Vulnerabilities");
if (vulnerabilitiesJSONArray == null || vulnerabilitiesJSONArray.size() == 0) {
vulnerabilitiesJSONArray = new JSONArray();
}
Vulnerability vulnerability;
for (Object v : vulnerabilitiesJSONArray) {
JSONObject vulnerabilitiesJSONObject = (JSONObject) v;
vulnerability = new Vulnerability();
String vulnerabilityName = vulnerabilitiesJSONObject.getString("Name");
String vulnerabilityNamespaceName = vulnerabilitiesJSONObject.getString("NamespaceName");
String vulnerabilityDescription = vulnerabilitiesJSONObject.getString("Description");
String vulnerabilityLink = vulnerabilitiesJSONObject.getString("Link");
String vulnerabilitySeverity = vulnerabilitiesJSONObject.getString("Severity");
String vulnerabilityFixedBy = vulnerabilitiesJSONObject.getString("FixedBy");
vulnerability.setName(vulnerabilityName);
vulnerability.setNamespaceName(vulnerabilityNamespaceName);
vulnerability.setDescription(vulnerabilityDescription);
vulnerability.setLink(vulnerabilityLink);
vulnerability.setSeverity(vulnerabilitySeverity);
vulnerability.setFixedBy(vulnerabilityFixedBy);
vulnerabilities.add(vulnerability);
}
feature.setName(name);
feature.setNamespaceName(namespaceName);
feature.setVersionFormat(versionFormat);
feature.setVersion(version);
feature.setAddedBy(addedBy);
feature.setVulnerabilities(vulnerabilities);
featureList.add(feature);
}
//4.封装镜像扫描结果
ImageScanResult imageScanResult = new ImageScanResult();
int featureNum = featureList.size();
long vulnerabilityNum = 0;
long high = 0;
long medium = 0;
long low = 0;
for (Feature temp : featureList) {
List<Vulnerability> vulnerabilityList = temp.getVulnerabilities();
vulnerabilityNum += vulnerabilityList.size();
high += vulnerabilityList.stream().filter(v -> v.getSeverity().equals("High")).count();
medium += vulnerabilityList.stream().filter(v -> v.getSeverity().equals("Medium")).count();
low += vulnerabilityList.stream().filter(v -> v.getSeverity().equals("Low")).count();
}
imageScanResult.setImageName(imageScan.getImageName());
imageScanResult.setImageTag(imageScan.getImageTag());
imageScanResult.setFeatureNum(featureNum);
imageScanResult.setVulnerabilityNum(vulnerabilityNum);
imageScanResult.setHigh(high);
imageScanResult.setMedium(medium);
imageScanResult.setLow(low);
imageScanResult.setFeatureList(featureList);
return imageScanResult;
}
（4）善后工作
清理扫描成功后的文件，这时我们打算清理解压出来的所有的tar包文件，否则解压目录将会异常庞大。博主在清理完一个24G的目录后，仅剩下几M的额外文件。
为什么我们不直接删除所有解压目录？这是因为我们需要留下镜像的分层描述文件（manifest.json），这个文件存放着该镜像的分层信息，包括分层id，分层tar包的地址等。后期我们可以依据此文件拿到最后一层的id，并调用clair api直接获取到该镜像的扫描结果。如果我们将扫描结果单独入库的话，那可以在扫描成功后删除整个解压目录。
//扫描完成后删除镜像解压目录
private void delImageDirAfterScan(ImageScan imageScan) {
String imageIdentification = formatImageNameAndTag(imageScan.getImageName(), imageScan.getImageTag());
String imageDir = systemConfig.getImageScanShellPath() + "scanImage/" + imageIdentification + "/";
SSH ssh = new SSH(systemConfig.getRelayServerIp(), systemConfig.getRelayServerUser(), systemConfig.getRelayServerPwd());
try {
boolean isConnect = ssh.connect();
if (isConnect) {
ssh.execute("find " + imageDir + " -name '*.tar' -type f -print -exec rm -rf {} \\;");
}
} catch (IOException e) {
e.printStackTrace();
}
}
京公网安备 11010502036488号