Milvus

概述Milvus是一个云原生向量数据库采用存算分离架构支持向量相似度与标量过滤的混合查询能在毫秒级完成十亿级向量检索并可按需选择强一致或最终一致等多种一致性等级。从业务角度milvus的数据模型层级是Database Collection Partition EntityDatabase数据库隔离不同业务数据Collection核心的逻辑容器类似mysql的tablePartition分区是collection的子集不是必须创建而且一个collection至少有默认partitionEntitycollection中的一条记录类似MySQL的一行数据1.部署和配置milvus部署方式很多支持大规模部署在k8s这里为了学习研究采用docker-compose安装Milvus以2.6.19版本为例docker-compose文件从官方github下载即可GitHub - milvus-io/milvus: Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search · GitHub启动前先修改docker-compose文件启用认证否则就算重置了密码仍然能免密登录找到milvus-standalone新增common.security.authorizationEnabled: true这个是2.6.x版本的做法... ... standalone: container_name: milvus-standalone image: milvusdb/milvus:v2.6.19 ... ... environment: ... ... common.security.authorizationEnabled: true ... ...然后docker compose启动即可启动后有三个容器还监听了不止一个端口[rootrocky102 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 47d071e552f2 milvusdb/milvus:v2.6.19 /tini -- milvus run… 25 minutes ago Up 25 minutes (healthy) 0.0.0.0:9091-9091/tcp, [::]:9091-9091/tcp, 0.0.0.0:19530-19530/tcp, [::]:19530-19530/tcp milvus-standalone a6e864be674d minio/minio:RELEASE.2024-12-18T13-15-44Z /usr/bin/docker-ent… 25 minutes ago Up 25 minutes (healthy) 0.0.0.0:9000-9001-9000-9001/tcp, [::]:9000-9001-9000-9001/tcp milvus-minio c4d5ee2d0bf6 quay.io/coreos/etcd:v3.5.25 etcd -advertise-cli… 25 minutes ago Up 25 minutes (healthy) 2379-2380/tcp milvus-etcd19530端口用于连接数据库9091端口还运行着webui9000-9001的minio也在运行以及etcd的2379为了安全建议除了19530端口外其余端口不要暴露然后修改用户名密码先用默认的用户名root密码Milvus登录进去然后马上修改密码连接milvus可以使用工具AttuGitHub - zilliztech/attu: The Best GUI for Milvus · GitHubattu应该也是使用某些web包壳工具开发而成每次数据库对象变动需要点击View-Reload按钮刷新才能实时展现2.Milvus Python API使用python操作milvus需安装pymilvus模块pip install pymilvus2.1 Database API2.1.1 连接还可通过MilvusClient的db_name参数指定一个库import os from dotenv import load_dotenv from pymilvus import MilvusClient, utility load_dotenv(encodingutf-8) MILVUS_HOST os.getenv(MILVUS_HOST) MILVUS_PORT os.getenv(MILVUS_PORT) MILVUS_KEY os.getenv(MILVUS_KEY) try: client MilvusClient( urifhttp://{MILVUS_HOST}:{MILVUS_PORT}, passwordMILVUS_KEY, userroot ) print(登录成功) except Exception as e: print(f{e})2.1.2 获取数据库dbs client.list_databases() for db in dbs: print(db)2.1.3 建库和删库创建dbs client.list_databases() name rag_demo if name not in dbs: client.create_database(name)删除如果下面有collection则无法删除需要先删除collectiondbs client.list_databases() name rag_demo if name in dbs: client.drop_database(db_namename)切换数据库name rag_demo client.use_database(name)2.2 Collection API2.2.1 获取Collectionuse_database执行后后续都默认在rag_demo下操作name rag_demo client.use_database(name) cols client.list_collections() print(cols)2.2.2 创建collection创建collection指定向量维度1024个采用余弦相似度name t_doc client.create_collection( collection_namename, dimension1024, metric_typeCOSINE )除此外还能设置auto_idTrue是否自增enable_dynamic_fieldFalse是否启用动态字段Collections 中的动态字段是一个保留的 JavaScript Object Notation (JSON) 字段名为$meta。启用该字段后Milvus 会将每个实体中携带的所有非 Schema 定义字段及其值作为键值对保存在保留字段中。2.2.3 删除collectionname t_doc client.drop_collection(collection_namename)2.2.4 描述信息元信息描述name t_doc metadata client.describe_collection(collection_namename) print(metadata){ collection_name: t_doc, auto_id: False, num_shards: 1, description: , fields: [ { field_id: 100, name: id, description: , type: DataType.INT64: 5, params: {}, is_primary: True }, { field_id: 101, name: vector, description: , type: DataType.FLOAT_VECTOR: 101, params: {dim: 1024} } ], functions: [], aliases: [], collection_id: 468082018063159696, consistency_level: 2, consistency_level_name: Bounded, properties: {timezone: UTC}, num_partitions: 1, enable_dynamic_field: True, enable_namespace: False, schema_version: 0, created_timestamp: 468092261983584273, update_timestamp: 468092261983584273 }2.2.5 判断存在name t_doc bmetadata client.has_collection(collection_namename) print(b)2.3 DML API关于数据的编辑2.3.1 添加数据通过维度同是1024的嵌入模型qwen3.7-text-embedding将一组文本转换为向量连同内容一起插入到milvus中并flush()落盘from dotenv import load_dotenv from openai import OpenAI import os from pymilvus import MilvusClient, utility load_dotenv(encodingutf-8) MILVUS_HOST os.getenv(MILVUS_HOST) MILVUS_PORT os.getenv(MILVUS_PORT) MILVUS_KEY os.getenv(MILVUS_KEY) client OpenAI( api_keyos.getenv(QWEN_KEY), base_urlhttps://llm-aryq90lbx0u4xri7.cn-beijing.maas.aliyuncs.com/compatible-mode/v1, ) texts [ 中国河北发生滦河第一号洪水, 菲律宾和中国就南海问题进行交涉, 武大靖被韩国人在ins上谩骂, 日本政府决定将核污染水进行排海, 中华人民共和国全运会在天津开幕, 在中国的调节下沙特和伊朗和解, 谷爱凌在2022北京冬奥会上获得滑雪冠军, 缅甸曼德勒发生8.0级地震, 无法忍受北约东扩俄罗斯进攻乌克兰, 湘潭大学周立人因投毒被判处死刑, 全红婵在东京奥运会获得跳水金牌 ] try: res client.embeddings.create( modelqwen3.7-text-embedding, inputtexts, ) data [ { id: i, vector: res.data[i].embedding, text: texts[i], source: demo } for i in range(len(texts)) ] client MilvusClient( urifhttp://{MILVUS_HOST}:{MILVUS_PORT}, passwordMILVUS_KEY, userroot, db_namerag_demo ) insert_res client.upsert( collection_namet_doc, datadata, ) print(insert result : , insert_res) client.flush(collection_namet_doc) except Exception as e: print(f{e})insert result : {upsert_count: 11, ids: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}2.3.2 状态name t_doc stats client.get_collection_stats(collection_namename) print(stats) #{row_count: 11}2.4 DQL API关于数据查询2.4.1 查询全部name t_doc iterator client.query_iterator( collection_namename, filter, output_fields[*] ) while True: rows iterator.next() if not rows: break for row in rows: print(fid : {row[id]},vector {row[vector][:5]}......,text {row[text]},source {row[source]}) iterator.close()id : 0,vector [0.023675184696912766, -0.005329845007508993, -0.010327795520424843, -0.007353754714131355, 0.03331966698169708]......,text 中国河北发生滦河第一号洪水,source demo id : 1,vector [0.03367263078689575, 0.0308539941906929, 0.0456014983355999, -0.05586938560009003, -0.04142387956380844]......,text 菲律宾和中国就南海问题进行交涉,source demo id : 2,vector [0.022217441350221634, 0.08715223520994186, 0.03312373906373978, 0.0017420633230358362, 0.014378155581653118]......,text 武大靖被韩国人在ins上谩骂,source demo id : 3,vector [0.026949338614940643, -0.025609316304326057, -0.011805844493210316, -0.06710037589073181, -0.01656416431069374]......,text 日本政府决定将核污染水进行排海,source demo id : 4,vector [0.023202229291200638, 0.03087700717151165, -0.040330708026885986, 9.698700159788132e-05, -0.03369786962866783]......,text 中华人民共和国全运会在天津开幕,source demo id : 5,vector [0.030343549326062202, 0.1002483069896698, -0.03368183597922325, -0.02625788189470768, -0.028873704373836517]......,text 在中国的调节下沙特和伊朗和解,source demo id : 6,vector [0.017193175852298737, 0.05799376219511032, -0.049061913043260574, 0.0675581768155098, -0.012638692744076252]......,text 谷爱凌在2022北京冬奥会上获得滑雪冠军,source demo id : 7,vector [0.03244605287909508, 0.015612950548529625, 0.04743647575378418, 0.00950597133487463, 0.023170415312051773]......,text 缅甸曼德勒发生8.0级地震,source demo id : 8,vector [0.02195076085627079, 0.07907696813344955, 0.034633148461580276, 0.00758601538836956, 0.020570363849401474]......,text 无法忍受北约东扩俄罗斯进攻乌克兰,source demo id : 9,vector [0.0225443746894598, -0.00823928415775299, -0.012978585436940193, -0.0132401492446661, 0.0005052244523540139]......,text 湘潭大学周立人因投毒被判处死刑,source demo id : 10,vector [0.023601844906806946, 0.0007014941074885428, 0.0010958664352074265, 0.06690911948680878, 0.03573780879378319]......,text 全红婵在东京奥运会获得跳水金牌,source demo2.4.2 根据IDname t_doc res client.get( collection_namename, ids[0,1] ) for i in range(len(res)): print(fid : {res[i][id]},vector {res[i][vector][:5]},text {res[i][text]},source {res[i][source]})2.4.3 相似度检索将待查关键字通过嵌入模型转化为向量传给milvus进行检索取出最匹配的前6个并指定返回字段:[text, source, id]from dotenv import load_dotenv from openai import OpenAI import os from pymilvus import MilvusClient, utility load_dotenv(encodingutf-8) MILVUS_HOST os.getenv(MILVUS_HOST) MILVUS_PORT os.getenv(MILVUS_PORT) MILVUS_KEY os.getenv(MILVUS_KEY) client OpenAI( api_keyos.getenv(QWEN_KEY), base_urlhttps://llm-aryq90lbx0u4xri7.cn-beijing.maas.aliyuncs.com/compatible-mode/v1, ) try: res client.embeddings.create( modelqwen3.7-text-embedding, input[体育赛事], ) vector res.data[0].embedding client MilvusClient( urifhttp://{MILVUS_HOST}:{MILVUS_PORT}, passwordMILVUS_KEY, userroot, db_namerag_demo ) results client.search( collection_namet_doc, data[vector], limit6, output_fields[text, source, id] ) for res in results[0]: print(res) except Exception as e: print(e){id: 4, distance: 0.4277065396308899, entity: {id: 4, text: 中华人民共和国全运会在天津开幕, source: demo}} {id: 2, distance: 0.2563920021057129, entity: {id: 2, text: 武大靖被韩国人在ins上谩骂, source: demo}} {id: 6, distance: 0.2262747436761856, entity: {id: 6, text: 谷爱凌在2022北京冬奥会上获得滑雪冠军, source: demo}} {id: 0, distance: 0.22332191467285156, entity: {id: 0, text: 中国河北发生滦河第一号洪水, source: demo}} {id: 5, distance: 0.18999835848808289, entity: {id: 5, text: 在中国的调节下沙特和伊朗和解, source: demo}} {id: 7, distance: 0.17732326686382294, entity: {id: 7, text: 缅甸曼德勒发生8.0级地震, source: demo}}